[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:35:25 dist-test-slave-tcgx krb5kdc[395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:35:25 dist-test-slave-tcgx krb5kdc[395](info): set up 2 sockets
Jun 20 23:35:25 dist-test-slave-tcgx krb5kdc[395](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 23:35:26.716856 388 mini_kdc.cc:121] Time spent starting KDC: real 1.118s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/krb5kdc/test-user.keytab.
Jun 20 23:35:26 dist-test-slave-tcgx krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:35:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:35:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab.
W20260620 23:35:27.087091 388 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.97.62: real 0.151s user 0.003s sys 0.005s
I20260620 23:35:27.088670 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46757
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46757
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:35:27.426815 408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:35:27.427205 408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:35:27.427311 408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:35:27.434634 408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:35:27.434751 408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:35:27.434798 408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:35:27.434862 408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:35:27.434900 408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:35:27.445408 408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46757
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:35:27.446769 408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:35:27.448235 408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:35:27.461078 416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:35:27.461119 414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:35:27.461324 413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:35:27.462015 408 server_base.cc:1034] running on GCE node
I20260620 23:35:27.462855 408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:35:27.464425 408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:35:27.465674 408 hybrid_clock.cc:648] HybridClock initialized: now 1781998527465556 us; error 111 us; skew 500 ppm
I20260620 23:35:27.466066 408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:35:27 dist-test-slave-tcgx krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998527, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:35:27.469197 408 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 23:35:27.470880 408 webserver.cc:460] Webserver started at http://127.0.97.62:46331/ using document root <none> and password file <none>
I20260620 23:35:27.471647 408 fs_manager.cc:359] Metadata directory not provided
I20260620 23:35:27.471793 408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:35:27.472102 408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:35:27.475181 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/minicluster-data/master-0/data/instance:
uuid: "a881760fd3494a258578ef5cdc190b45"
format_stamp: "Formatted at 2026-06-20 23:35:27 on dist-test-slave-tcgx"
server_key: "ec81d37b3db02557cb59a93b4d320ac6"
server_key_iv: "7fe9bd3d2b9921dc08ad80db74bafca8"
server_key_version: "encryptionkey@0"
I20260620 23:35:27.475965 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/minicluster-data/master-0/wal/instance:
uuid: "a881760fd3494a258578ef5cdc190b45"
format_stamp: "Formatted at 2026-06-20 23:35:27 on dist-test-slave-tcgx"
server_key: "ec81d37b3db02557cb59a93b4d320ac6"
server_key_iv: "7fe9bd3d2b9921dc08ad80db74bafca8"
server_key_version: "encryptionkey@0"
I20260620 23:35:27.482437 408 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 23:35:27.487039 423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:35:27.488482 408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260620 23:35:27.488687 408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/minicluster-data/master-0/wal
uuid: "a881760fd3494a258578ef5cdc190b45"
format_stamp: "Formatted at 2026-06-20 23:35:27 on dist-test-slave-tcgx"
server_key: "ec81d37b3db02557cb59a93b4d320ac6"
server_key_iv: "7fe9bd3d2b9921dc08ad80db74bafca8"
server_key_version: "encryptionkey@0"
I20260620 23:35:27.488898 408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:35:27.503681 408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:35:27.508270 408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:35:27.508598 408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:35:27.530166 408 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46757
I20260620 23:35:27.530189 474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46757 every 8 connection(s)
I20260620 23:35:27.531735 408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:35:27.535707 475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:35:27.536952 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 408
I20260620 23:35:27.537232 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/minicluster-data/master-0/wal/instance
I20260620 23:35:27.539541 388 external_mini_cluster.cc:1428] Setting key c6abf951179a0f7de1738311671820ec
I20260620 23:35:27.546811 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a881760fd3494a258578ef5cdc190b45: Bootstrap starting.
I20260620 23:35:27.549852 475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a881760fd3494a258578ef5cdc190b45: Neither blocks nor log segments found. Creating new log.
I20260620 23:35:27.550921 475 log.cc:824] T 00000000000000000000000000000000 P a881760fd3494a258578ef5cdc190b45: Log is configured to *not* fsync() on all Append() calls
I20260620 23:35:27.554191 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a881760fd3494a258578ef5cdc190b45: No bootstrap required, opened a new log
I20260620 23:35:27.563970 475 raft_consensus.cc:348] T 00000000000000000000000000000000 P a881760fd3494a258578ef5cdc190b45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a881760fd3494a258578ef5cdc190b45" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46757 } }
I20260620 23:35:27.564462 475 raft_consensus.cc:374] T 00000000000000000000000000000000 P a881760fd3494a258578ef5cdc190b45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:35:27.564581 475 raft_consensus.cc:729] T 00000000000000000000000000000000 P a881760fd3494a258578ef5cdc190b45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a881760fd3494a258578ef5cdc190b45, State: Initialized, Role: FOLLOWER
I20260620 23:35:27.565335 475 consensus_queue.cc:260] T 00000000000000000000000000000000 P a881760fd3494a258578ef5cdc190b45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a881760fd3494a258578ef5cdc190b45" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46757 } }
I20260620 23:35:27.565640 475 raft_consensus.cc:388] T 00000000000000000000000000000000 P a881760fd3494a258578ef5cdc190b45 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:35:27.565801 475 raft_consensus.cc:482] T 00000000000000000000000000000000 P a881760fd3494a258578ef5cdc190b45 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:35:27.565976 475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a881760fd3494a258578ef5cdc190b45 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:35:27.568542 475 raft_consensus.cc:504] T 00000000000000000000000000000000 P a881760fd3494a258578ef5cdc190b45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a881760fd3494a258578ef5cdc190b45" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46757 } }
I20260620 23:35:27.569144 475 leader_election.cc:302] T 00000000000000000000000000000000 P a881760fd3494a258578ef5cdc190b45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a881760fd3494a258578ef5cdc190b45; no voters:
I20260620 23:35:27.569816 475 leader_election.cc:288] T 00000000000000000000000000000000 P a881760fd3494a258578ef5cdc190b45 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:35:27.569931 480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a881760fd3494a258578ef5cdc190b45 [term 1 FOLLOWER]: Leader election won for term 1
Jun 20 23:35:27 dist-test-slave-tcgx krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:35:27.572654 475 sys_catalog.cc:564] T 00000000000000000000000000000000 P a881760fd3494a258578ef5cdc190b45 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:35:27.572712 480 raft_consensus.cc:686] T 00000000000000000000000000000000 P a881760fd3494a258578ef5cdc190b45 [term 1 LEADER]: Becoming Leader. State: Replica: a881760fd3494a258578ef5cdc190b45, State: Running, Role: LEADER
I20260620 23:35:27.573339 480 consensus_queue.cc:237] T 00000000000000000000000000000000 P a881760fd3494a258578ef5cdc190b45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a881760fd3494a258578ef5cdc190b45" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46757 } }
I20260620 23:35:27.579638 482 sys_catalog.cc:455] T 00000000000000000000000000000000 P a881760fd3494a258578ef5cdc190b45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a881760fd3494a258578ef5cdc190b45. Latest consensus state: current_term: 1 leader_uuid: "a881760fd3494a258578ef5cdc190b45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a881760fd3494a258578ef5cdc190b45" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46757 } } }
I20260620 23:35:27.580108 482 sys_catalog.cc:458] T 00000000000000000000000000000000 P a881760fd3494a258578ef5cdc190b45 [sys.catalog]: This master's current role is: LEADER
I20260620 23:35:27.580688 481 sys_catalog.cc:455] T 00000000000000000000000000000000 P a881760fd3494a258578ef5cdc190b45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a881760fd3494a258578ef5cdc190b45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a881760fd3494a258578ef5cdc190b45" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46757 } } }
I20260620 23:35:27.581008 481 sys_catalog.cc:458] T 00000000000000000000000000000000 P a881760fd3494a258578ef5cdc190b45 [sys.catalog]: This master's current role is: LEADER
I20260620 23:35:27.589269 477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:27.543544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33792 (local address 127.0.97.62:46757)
0620 23:35:27.544560 (+ 1016us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:27.544575 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:27.545870 (+ 1295us) server_negotiation.cc:407] Connection header received
0620 23:35:27.555769 (+ 9899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:27.555814 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:27.556365 (+ 551us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:27.556870 (+ 505us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:35:27.559967 (+ 3097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:27.562105 (+ 2138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:27.564025 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:27.565078 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:27.565258 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:27.572649 (+ 7391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:35:27.572663 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:35:27.572689 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:35:27.572762 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:35:27.577775 (+ 5013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:27.579141 (+ 1366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:27.579152 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:27.579160 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:27.579386 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:27.580014 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:27.580018 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:27.580022 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:27.581261 (+ 1239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:35:27.582123 (+ 862us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:27.586796 (+ 4673us) server_negotiation.cc:299] Negotiation successful
0620 23:35:27.587474 (+ 678us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":721,"thread_start_us":449,"threads_started":1}
I20260620 23:35:27.589884 489 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:35:27.595660 489 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:35:27.606683 489 catalog_manager.cc:1347] Generated new cluster ID: bb9284e0773440b19707dcd82fd174e7
I20260620 23:35:27.606827 489 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:35:27.631271 489 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:35:27.632723 489 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:35:27.640705 489 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a881760fd3494a258578ef5cdc190b45: Generated new TSK 0
I20260620 23:35:27.642040 489 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 23:35:27.692494 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46757
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:35:28.056823 502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:35:28.057160 502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:35:28.057329 502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:35:28.064757 502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:35:28.064918 502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:35:28.065097 502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 23:35:28.075722 502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46757
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:35:28.077095 502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:35:28.078501 502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:35:28.092036 510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:35:28.092276 508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:35:28.092455 507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:35:28.093758 502 server_base.cc:1034] running on GCE node
I20260620 23:35:28.094493 502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:35:28.095566 502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:35:28.096761 502 hybrid_clock.cc:648] HybridClock initialized: now 1781998528096673 us; error 82 us; skew 500 ppm
I20260620 23:35:28.097085 502 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:35:28 dist-test-slave-tcgx krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:35:28.100620 502 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 23:35:28.102355 502 webserver.cc:460] Webserver started at http://127.0.97.1:39027/ using document root <none> and password file <none>
I20260620 23:35:28.103178 502 fs_manager.cc:359] Metadata directory not provided
I20260620 23:35:28.103339 502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:35:28.103668 502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:35:28.106805 502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/minicluster-data/ts-0/data/instance:
uuid: "6881d34a062d45e3af0274c97956f6c7"
format_stamp: "Formatted at 2026-06-20 23:35:28 on dist-test-slave-tcgx"
server_key: "88a28afef0efd84dc3647102f5bf5748"
server_key_iv: "541d478843524b3e9abe27cc8c0ec8a0"
server_key_version: "encryptionkey@0"
I20260620 23:35:28.107759 502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/minicluster-data/ts-0/wal/instance:
uuid: "6881d34a062d45e3af0274c97956f6c7"
format_stamp: "Formatted at 2026-06-20 23:35:28 on dist-test-slave-tcgx"
server_key: "88a28afef0efd84dc3647102f5bf5748"
server_key_iv: "541d478843524b3e9abe27cc8c0ec8a0"
server_key_version: "encryptionkey@0"
I20260620 23:35:28.113428 502 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:35:28.117275 517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:35:28.118690 502 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260620 23:35:28.118888 502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/minicluster-data/ts-0/wal
uuid: "6881d34a062d45e3af0274c97956f6c7"
format_stamp: "Formatted at 2026-06-20 23:35:28 on dist-test-slave-tcgx"
server_key: "88a28afef0efd84dc3647102f5bf5748"
server_key_iv: "541d478843524b3e9abe27cc8c0ec8a0"
server_key_version: "encryptionkey@0"
I20260620 23:35:28.119139 502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:35:28.143108 502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:35:28.147204 502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:35:28.147490 502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:35:28.148464 502 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:35:28.149775 502 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:35:28.149887 502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:35:28.150029 502 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:35:28.150100 502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:35:28.186573 502 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33603
I20260620 23:35:28.186627 629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33603 every 8 connection(s)
I20260620 23:35:28.187940 502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
I20260620 23:35:28.192987 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 502
I20260620 23:35:28.193207 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/minicluster-data/ts-0/wal/instance
I20260620 23:35:28.193893 388 external_mini_cluster.cc:1428] Setting key a288a0d4dac5f267e94e5b28df957d62
Jun 20 23:35:28 dist-test-slave-tcgx krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 23:35:28.212021 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:28.190904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42111 (local address 127.0.97.62:46757)
0620 23:35:28.191335 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:28.191340 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:28.192498 (+ 1158us) server_negotiation.cc:407] Connection header received
0620 23:35:28.194634 (+ 2136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:28.194639 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:28.194739 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:28.194877 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:35:28.196721 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:28.197427 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:28.199069 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:28.199800 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:28.199958 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:28.204808 (+ 4850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:35:28.204814 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:35:28.204818 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:35:28.204856 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:35:28.208634 (+ 3778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:28.209517 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:28.209524 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:28.209529 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:28.209687 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:28.210140 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:28.210146 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:28.210149 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:28.210536 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:35:28.210700 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:28.211598 (+ 898us) server_negotiation.cc:299] Negotiation successful
0620 23:35:28.211765 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":232,"threads_started":1}
I20260620 23:35:28.213025 633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:28.191581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46757 (local address 127.0.97.1:42111)
0620 23:35:28.192293 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0620 23:35:28.192369 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0620 23:35:28.194314 (+ 1945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:35:28.195174 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:35:28.195186 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:35:28.195966 (+ 780us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:35:28.196552 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:28.196569 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:28.197581 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:28.197586 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:28.198827 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:28.198839 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:28.200064 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:28.200072 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:28.200413 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:28.201272 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:35:28.201306 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:35:28.204535 (+ 3229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:35:28.208829 (+ 4294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:35:28.208838 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:35:28.208854 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:35:28.209340 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:35:28.209815 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:35:28.209819 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:35:28.209821 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:35:28.210010 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:35:28.210677 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:35:28.210683 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:35:28.211144 (+ 461us) client_negotiation.cc:769] Sending connection context
0620 23:35:28.211421 (+ 277us) client_negotiation.cc:240] Negotiation successful
0620 23:35:28.211843 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":272,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260620 23:35:28.214720 630 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46757
I20260620 23:35:28.215073 630 heartbeater.cc:464] Registering TS with master...
I20260620 23:35:28.215801 630 heartbeater.cc:511] Master 127.0.97.62:46757 requested a full tablet report, sending...
I20260620 23:35:28.217979 440 ts_manager.cc:194] Registered new tserver with Master: 6881d34a062d45e3af0274c97956f6c7 (127.0.97.1:33603)
I20260620 23:35:28.219719 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:42111
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 23:35:28.238101 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46757
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:35:28.586419 636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:35:28.586774 636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:35:28.586987 636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:35:28.595432 636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:35:28.595582 636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:35:28.595769 636 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 23:35:28.605794 636 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46757
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:35:28.607354 636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:35:28.608691 636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:35:28.622828 644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:35:28.622901 641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:35:28.622917 642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:35:28.624046 636 server_base.cc:1034] running on GCE node
I20260620 23:35:28.624784 636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:35:28.625833 636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:35:28.627061 636 hybrid_clock.cc:648] HybridClock initialized: now 1781998528626986 us; error 69 us; skew 500 ppm
I20260620 23:35:28.627432 636 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:35:28 dist-test-slave-tcgx krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:35:28.630771 636 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 23:35:28.632411 636 webserver.cc:460] Webserver started at http://127.0.97.2:37323/ using document root <none> and password file <none>
I20260620 23:35:28.633104 636 fs_manager.cc:359] Metadata directory not provided
I20260620 23:35:28.633204 636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:35:28.633458 636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:35:28.636503 636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/minicluster-data/ts-1/data/instance:
uuid: "83e8656b20614102a7d7f35361129c19"
format_stamp: "Formatted at 2026-06-20 23:35:28 on dist-test-slave-tcgx"
server_key: "71c015246bee549141e7b3146784b614"
server_key_iv: "9f6115318953f81f4d7a4c758ab40016"
server_key_version: "encryptionkey@0"
I20260620 23:35:28.637316 636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/minicluster-data/ts-1/wal/instance:
uuid: "83e8656b20614102a7d7f35361129c19"
format_stamp: "Formatted at 2026-06-20 23:35:28 on dist-test-slave-tcgx"
server_key: "71c015246bee549141e7b3146784b614"
server_key_iv: "9f6115318953f81f4d7a4c758ab40016"
server_key_version: "encryptionkey@0"
I20260620 23:35:28.643118 636 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 23:35:28.646783 651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:35:28.648361 636 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:35:28.648514 636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/minicluster-data/ts-1/wal
uuid: "83e8656b20614102a7d7f35361129c19"
format_stamp: "Formatted at 2026-06-20 23:35:28 on dist-test-slave-tcgx"
server_key: "71c015246bee549141e7b3146784b614"
server_key_iv: "9f6115318953f81f4d7a4c758ab40016"
server_key_version: "encryptionkey@0"
I20260620 23:35:28.648694 636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:35:28.677078 636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:35:28.682384 636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:35:28.682670 636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:35:28.683795 636 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:35:28.685144 636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:35:28.685220 636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:35:28.685304 636 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:35:28.685352 636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:35:28.721397 636 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41541
I20260620 23:35:28.721444 763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41541 every 8 connection(s)
I20260620 23:35:28.722735 636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
I20260620 23:35:28.726240 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 636
I20260620 23:35:28.726434 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/minicluster-data/ts-1/wal/instance
I20260620 23:35:28.727051 388 external_mini_cluster.cc:1428] Setting key 5bea3f0e41c47ebb6bcd993e4dae9c3e
Jun 20 23:35:28 dist-test-slave-tcgx krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260620 23:35:28.748170 766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:28.725506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35063 (local address 127.0.97.62:46757)
0620 23:35:28.725938 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:28.725942 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:28.727092 (+ 1150us) server_negotiation.cc:407] Connection header received
0620 23:35:28.729442 (+ 2350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:28.729447 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:28.729534 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:28.729645 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:35:28.731202 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:28.731913 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:28.733891 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:28.734298 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:28.734433 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:28.741695 (+ 7262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:35:28.741701 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:35:28.741705 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:35:28.741742 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:35:28.744333 (+ 2591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:28.745357 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:28.745365 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:28.745371 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:28.745537 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:28.746006 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:28.746015 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:28.746022 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:28.746420 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:35:28.746586 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:28.747396 (+ 810us) server_negotiation.cc:299] Negotiation successful
0620 23:35:28.747668 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":231,"threads_started":1}
I20260620 23:35:28.748864 767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:28.726066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46757 (local address 127.0.97.2:35063)
0620 23:35:28.726886 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0620 23:35:28.726923 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 23:35:28.729188 (+ 2265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:35:28.729889 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:35:28.729897 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:35:28.730397 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:35:28.731046 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:28.731065 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:28.732054 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:28.732059 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:28.733699 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:28.733711 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:28.734533 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:28.734540 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:28.734768 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:28.739106 (+ 4338us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:35:28.739134 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:35:28.741401 (+ 2267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:35:28.744575 (+ 3174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:35:28.744588 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:35:28.744612 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:35:28.745200 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:35:28.745658 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:35:28.745661 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:35:28.745664 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:35:28.745874 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:35:28.746559 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:35:28.746565 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:35:28.747007 (+ 442us) client_negotiation.cc:769] Sending connection context
0620 23:35:28.747256 (+ 249us) client_negotiation.cc:240] Negotiation successful
0620 23:35:28.747666 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":355,"threads_started":1}
I20260620 23:35:28.750566 764 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46757
I20260620 23:35:28.750880 764 heartbeater.cc:464] Registering TS with master...
I20260620 23:35:28.751642 764 heartbeater.cc:511] Master 127.0.97.62:46757 requested a full tablet report, sending...
I20260620 23:35:28.753240 440 ts_manager.cc:194] Registered new tserver with Master: 83e8656b20614102a7d7f35361129c19 (127.0.97.2:41541)
I20260620 23:35:28.754582 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:35063
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 23:35:28.770018 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46757
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:35:29.105139 770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:35:29.105448 770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:35:29.105607 770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:35:29.112969 770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:35:29.113098 770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:35:29.113669 770 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 23:35:29.123896 770 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46757
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:35:29.125399 770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:35:29.126801 770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:35:29.141848 775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:35:29.141793 778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:35:29.141736 776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:35:29.142895 770 server_base.cc:1034] running on GCE node
I20260620 23:35:29.143633 770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:35:29.144668 770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:35:29.145867 770 hybrid_clock.cc:648] HybridClock initialized: now 1781998529145798 us; error 69 us; skew 500 ppm
I20260620 23:35:29.146221 770 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:35:29 dist-test-slave-tcgx krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:35:29.149603 770 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 23:35:29.151356 770 webserver.cc:460] Webserver started at http://127.0.97.3:34215/ using document root <none> and password file <none>
I20260620 23:35:29.152112 770 fs_manager.cc:359] Metadata directory not provided
I20260620 23:35:29.152262 770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:35:29.152596 770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:35:29.156160 770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/minicluster-data/ts-2/data/instance:
uuid: "b4f80e807cb948e799a6ef59aac73052"
format_stamp: "Formatted at 2026-06-20 23:35:29 on dist-test-slave-tcgx"
server_key: "27c302ce296d23889b3a4e8cd8701c61"
server_key_iv: "9a6df3e73d3ca9bfd2d2519b48b7fd28"
server_key_version: "encryptionkey@0"
I20260620 23:35:29.157015 770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/minicluster-data/ts-2/wal/instance:
uuid: "b4f80e807cb948e799a6ef59aac73052"
format_stamp: "Formatted at 2026-06-20 23:35:29 on dist-test-slave-tcgx"
server_key: "27c302ce296d23889b3a4e8cd8701c61"
server_key_iv: "9a6df3e73d3ca9bfd2d2519b48b7fd28"
server_key_version: "encryptionkey@0"
I20260620 23:35:29.162487 770 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 23:35:29.166195 785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:35:29.167759 770 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 23:35:29.167963 770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/minicluster-data/ts-2/wal
uuid: "b4f80e807cb948e799a6ef59aac73052"
format_stamp: "Formatted at 2026-06-20 23:35:29 on dist-test-slave-tcgx"
server_key: "27c302ce296d23889b3a4e8cd8701c61"
server_key_iv: "9a6df3e73d3ca9bfd2d2519b48b7fd28"
server_key_version: "encryptionkey@0"
I20260620 23:35:29.168188 770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:35:29.180576 770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:35:29.185344 770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:35:29.185660 770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:35:29.187254 770 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:35:29.188609 770 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:35:29.188716 770 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:35:29.188874 770 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:35:29.188944 770 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:35:29.222756 630 heartbeater.cc:503] Master 127.0.97.62:46757 was elected leader, sending a full tablet report...
I20260620 23:35:29.226073 770 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33341
I20260620 23:35:29.226119 897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33341 every 8 connection(s)
I20260620 23:35:29.227463 770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
I20260620 23:35:29.237293 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 770
I20260620 23:35:29.237479 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998525501163-388-0/minicluster-data/ts-2/wal/instance
I20260620 23:35:29.237900 388 external_mini_cluster.cc:1428] Setting key 0de928e4034709a2b11064a6f25a364b
Jun 20 23:35:29 dist-test-slave-tcgx krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:35:29.249184 766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:29.230169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56891 (local address 127.0.97.62:46757)
0620 23:35:29.230337 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:29.230342 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:29.231579 (+ 1237us) server_negotiation.cc:407] Connection header received
0620 23:35:29.233933 (+ 2354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:29.233939 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:29.234040 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:29.234154 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:35:29.235876 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:29.236637 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:29.238417 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:29.238843 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:29.239018 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:29.243381 (+ 4363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:35:29.243386 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:35:29.243390 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:35:29.243439 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:35:29.246093 (+ 2654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:29.246825 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:29.246831 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:29.246835 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:29.247063 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:29.247558 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:29.247571 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:29.247573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:29.247885 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:35:29.248023 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:29.248757 (+ 734us) server_negotiation.cc:299] Negotiation successful
0620 23:35:29.248947 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 23:35:29.250463 900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:29.230612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46757 (local address 127.0.97.3:56891)
0620 23:35:29.231410 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0620 23:35:29.231449 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 23:35:29.233620 (+ 2171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:35:29.234419 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:35:29.234429 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:35:29.235070 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:35:29.235674 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:29.235691 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:29.236798 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:29.236809 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:29.238257 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:29.238270 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:29.239110 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:29.239116 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:29.239440 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:29.240390 (+ 950us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:35:29.240430 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:35:29.243188 (+ 2758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:35:29.246221 (+ 3033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:35:29.246228 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:35:29.246241 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:35:29.246667 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:35:29.247217 (+ 550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:35:29.247221 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:35:29.247224 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:35:29.247418 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:35:29.248012 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:35:29.248019 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:35:29.248362 (+ 343us) client_negotiation.cc:769] Sending connection context
0620 23:35:29.248615 (+ 253us) client_negotiation.cc:240] Negotiation successful
0620 23:35:29.249056 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":368,"threads_started":1}
I20260620 23:35:29.252079 898 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46757
I20260620 23:35:29.252485 898 heartbeater.cc:464] Registering TS with master...
I20260620 23:35:29.253441 898 heartbeater.cc:511] Master 127.0.97.62:46757 requested a full tablet report, sending...
I20260620 23:35:29.255110 440 ts_manager.cc:194] Registered new tserver with Master: b4f80e807cb948e799a6ef59aac73052 (127.0.97.3:33341)
I20260620 23:35:29.255884 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:56891
I20260620 23:35:29.267282 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 23:35:29 dist-test-slave-tcgx krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998529, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 23:35:29 dist-test-slave-tcgx krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998529, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 23:35:29.314497 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:29.298157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46364 (local address 127.0.97.1:33603)
0620 23:35:29.298821 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:29.298828 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:29.298858 (+ 30us) server_negotiation.cc:407] Connection header received
0620 23:35:29.299014 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:29.299022 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:29.299241 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:29.299392 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:35:29.300768 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:29.301831 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:29.303152 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:29.303716 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:29.303882 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:29.308400 (+ 4518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:35:29.308413 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:35:29.308429 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:35:29.308467 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:35:29.311870 (+ 3403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:29.312590 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:29.312597 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:29.312599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:29.312730 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:29.313085 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:29.313088 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:29.313090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:29.313426 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:35:29.313640 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:29.313881 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 23:35:29.314078 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":382,"threads_started":1}
W20260620 23:35:29.316015 545 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46364
Jun 20 23:35:29 dist-test-slave-tcgx krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 23:35:29 dist-test-slave-tcgx krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 23:35:29.360517 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:29.346762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46370 (local address 127.0.97.1:33603)
0620 23:35:29.347029 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:29.347036 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:29.347286 (+ 250us) server_negotiation.cc:407] Connection header received
0620 23:35:29.347435 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:29.347439 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:29.347534 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:29.347624 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:35:29.349000 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:29.349779 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:29.351040 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:29.351470 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:29.351617 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:29.355568 (+ 3951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:35:29.355573 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:35:29.355576 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:35:29.355612 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:35:29.358245 (+ 2633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:29.358860 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:29.358864 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:29.358867 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:29.359078 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:29.359431 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:29.359434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:29.359436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:29.359680 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:35:29.359848 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:29.360104 (+ 256us) server_negotiation.cc:299] Negotiation successful
0620 23:35:29.360266 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 23:35:29.362056 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 502
I20260620 23:35:29.375851 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:29.363172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46376 (local address 127.0.97.1:33603)
0620 23:35:29.363375 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:29.363381 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:29.363572 (+ 191us) server_negotiation.cc:407] Connection header received
0620 23:35:29.363757 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:29.363762 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:29.363839 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:29.363935 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:35:29.365111 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:29.366219 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:29.367518 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:29.367933 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:29.368064 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:29.370345 (+ 2281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:35:29.370351 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:35:29.370355 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:35:29.370397 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:35:29.373210 (+ 2813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:29.373869 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:29.373880 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:29.373882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:29.374054 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:29.374442 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:29.374447 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:29.374451 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:29.374707 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:35:29.374979 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:29.375277 (+ 298us) server_negotiation.cc:299] Negotiation successful
0620 23:35:29.375502 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 23:35:29.376680 625 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:35:29.464226 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 502
I20260620 23:35:29.489219 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 636
Jun 20 23:35:29 dist-test-slave-tcgx krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 23:35:29.512645 912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:29.490818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40398 (local address 127.0.97.2:41541)
0620 23:35:29.491611 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:29.491621 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:29.491653 (+ 32us) server_negotiation.cc:407] Connection header received
0620 23:35:29.491732 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:29.491741 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:29.492131 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:29.492329 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:35:29.494232 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:29.495442 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:29.497738 (+ 2296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:29.498380 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:29.498497 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:29.504358 (+ 5861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:35:29.504380 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:35:29.504396 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:35:29.504447 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:35:29.508383 (+ 3936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:29.509539 (+ 1156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:29.509552 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:29.509557 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:29.509747 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:29.510306 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:29.510313 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:29.510319 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:29.511078 (+ 759us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:35:29.511389 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:29.511842 (+ 453us) server_negotiation.cc:299] Negotiation successful
0620 23:35:29.512232 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":402,"threads_started":1}
I20260620 23:35:29.514994 759 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:35:29.639436 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 636
I20260620 23:35:29.673446 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 770
Jun 20 23:35:29 dist-test-slave-tcgx krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 23:35:29.690414 914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:29.674566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33402 (local address 127.0.97.3:33341)
0620 23:35:29.675242 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:29.675249 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:29.675281 (+ 32us) server_negotiation.cc:407] Connection header received
0620 23:35:29.675339 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:29.675345 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:29.675567 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:29.675747 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:35:29.677274 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:29.678275 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:29.679763 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:29.680343 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:29.680501 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:29.684818 (+ 4317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:35:29.684831 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:35:29.684847 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:35:29.684891 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:35:29.687631 (+ 2740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:29.688293 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:29.688301 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:29.688304 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:29.688425 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:29.688802 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:29.688807 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:29.688810 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:29.689158 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:35:29.689464 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:29.689779 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 23:35:29.690055 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":396,"threads_started":1}
I20260620 23:35:29.691777 893 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:35:29.774425 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 770
I20260620 23:35:29.790768 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 408
I20260620 23:35:29.792068 470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:35:29.895665 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 408
2026-06-20T23:35:29Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4351 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:35:29 dist-test-slave-tcgx krb5kdc[919](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:35:29 dist-test-slave-tcgx krb5kdc[919](info): set up 2 sockets
Jun 20 23:35:29 dist-test-slave-tcgx krb5kdc[919](info): commencing operation
krb5kdc: starting...
W20260620 23:35:30.975344 388 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/krb5kdc/test-user.keytab.
Jun 20 23:35:31 dist-test-slave-tcgx krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:35:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:35:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 23:35:31.156479 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45035
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45035
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:35:31.481212 932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:35:31.481587 932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:35:31.481768 932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:35:31.489008 932 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:35:31.489178 932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:35:31.489265 932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:35:31.489367 932 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:35:31.489435 932 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:35:31.501027 932 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:35:31.502441 932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:35:31.503854 932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:35:31.517835 940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:35:31.517848 938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:35:31.518199 937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:35:31.519084 932 server_base.cc:1034] running on GCE node
I20260620 23:35:31.519928 932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:35:31.521420 932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:35:31.522657 932 hybrid_clock.cc:648] HybridClock initialized: now 1781998531522599 us; error 48 us; skew 500 ppm
I20260620 23:35:31.523103 932 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:35:31 dist-test-slave-tcgx krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998531, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:35:31.526247 932 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 23:35:31.527944 932 webserver.cc:460] Webserver started at http://127.0.97.62:35615/ using document root <none> and password file <none>
I20260620 23:35:31.528625 932 fs_manager.cc:359] Metadata directory not provided
I20260620 23:35:31.528725 932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:35:31.528987 932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:35:31.531996 932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/minicluster-data/master-0/data/instance:
uuid: "8831cb22bf6d4dc2abd40828c327677f"
format_stamp: "Formatted at 2026-06-20 23:35:31 on dist-test-slave-tcgx"
server_key: "0ee8c2e768ed751e8366c517b47c2d33"
server_key_iv: "344b3b93c437610941c52e8d3ebe0af3"
server_key_version: "encryptionkey@0"
I20260620 23:35:31.532701 932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/minicluster-data/master-0/wal/instance:
uuid: "8831cb22bf6d4dc2abd40828c327677f"
format_stamp: "Formatted at 2026-06-20 23:35:31 on dist-test-slave-tcgx"
server_key: "0ee8c2e768ed751e8366c517b47c2d33"
server_key_iv: "344b3b93c437610941c52e8d3ebe0af3"
server_key_version: "encryptionkey@0"
I20260620 23:35:31.538220 932 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260620 23:35:31.541719 947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:35:31.543215 932 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:35:31.543370 932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/minicluster-data/master-0/wal
uuid: "8831cb22bf6d4dc2abd40828c327677f"
format_stamp: "Formatted at 2026-06-20 23:35:31 on dist-test-slave-tcgx"
server_key: "0ee8c2e768ed751e8366c517b47c2d33"
server_key_iv: "344b3b93c437610941c52e8d3ebe0af3"
server_key_version: "encryptionkey@0"
I20260620 23:35:31.543545 932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:35:31.559015 932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:35:31.562929 932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:35:31.563190 932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:35:31.581928 932 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45035
I20260620 23:35:31.581954 998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45035 every 8 connection(s)
I20260620 23:35:31.583403 932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:35:31.587291 999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:35:31.593488 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 932
I20260620 23:35:31.593708 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/minicluster-data/master-0/wal/instance
I20260620 23:35:31.594547 388 external_mini_cluster.cc:1428] Setting key 24c2e8cd42c75f34a94cef3d9e560719
I20260620 23:35:31.598766 999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8831cb22bf6d4dc2abd40828c327677f: Bootstrap starting.
I20260620 23:35:31.602639 999 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8831cb22bf6d4dc2abd40828c327677f: Neither blocks nor log segments found. Creating new log.
I20260620 23:35:31.603855 999 log.cc:824] T 00000000000000000000000000000000 P 8831cb22bf6d4dc2abd40828c327677f: Log is configured to *not* fsync() on all Append() calls
I20260620 23:35:31.607021 999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8831cb22bf6d4dc2abd40828c327677f: No bootstrap required, opened a new log
Jun 20 23:35:31 dist-test-slave-tcgx krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:35:31.615329 999 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8831cb22bf6d4dc2abd40828c327677f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8831cb22bf6d4dc2abd40828c327677f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45035 } }
I20260620 23:35:31.615743 999 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8831cb22bf6d4dc2abd40828c327677f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:35:31.615864 999 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8831cb22bf6d4dc2abd40828c327677f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8831cb22bf6d4dc2abd40828c327677f, State: Initialized, Role: FOLLOWER
I20260620 23:35:31.616595 999 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8831cb22bf6d4dc2abd40828c327677f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8831cb22bf6d4dc2abd40828c327677f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45035 } }
I20260620 23:35:31.616916 999 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8831cb22bf6d4dc2abd40828c327677f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:35:31.617062 999 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8831cb22bf6d4dc2abd40828c327677f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:35:31.617256 999 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8831cb22bf6d4dc2abd40828c327677f [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:35:31.618451 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:31.596845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51480 (local address 127.0.97.62:45035)
0620 23:35:31.597668 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:31.597680 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:31.597724 (+ 44us) server_negotiation.cc:407] Connection header received
0620 23:35:31.599575 (+ 1851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:31.599599 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:31.600019 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:31.600389 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:35:31.602170 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:31.603601 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:31.605179 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:31.605853 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:31.605991 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:31.610595 (+ 4604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:35:31.610604 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:35:31.610619 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:35:31.610659 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:35:31.613936 (+ 3277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:31.614673 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:31.614681 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:31.614689 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:31.614854 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:31.615258 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:31.615264 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:31.615267 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:31.615915 (+ 648us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:35:31.616386 (+ 471us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:31.616763 (+ 377us) server_negotiation.cc:299] Negotiation successful
0620 23:35:31.617310 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":398,"threads_started":1}
I20260620 23:35:31.619772 999 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8831cb22bf6d4dc2abd40828c327677f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8831cb22bf6d4dc2abd40828c327677f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45035 } }
I20260620 23:35:31.620293 999 leader_election.cc:302] T 00000000000000000000000000000000 P 8831cb22bf6d4dc2abd40828c327677f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8831cb22bf6d4dc2abd40828c327677f; no voters:
I20260620 23:35:31.620878 999 leader_election.cc:288] T 00000000000000000000000000000000 P 8831cb22bf6d4dc2abd40828c327677f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:35:31.621083 1004 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8831cb22bf6d4dc2abd40828c327677f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:35:31.622217 1004 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8831cb22bf6d4dc2abd40828c327677f [term 1 LEADER]: Becoming Leader. State: Replica: 8831cb22bf6d4dc2abd40828c327677f, State: Running, Role: LEADER
I20260620 23:35:31.622776 1004 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8831cb22bf6d4dc2abd40828c327677f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8831cb22bf6d4dc2abd40828c327677f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45035 } }
I20260620 23:35:31.623567 999 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8831cb22bf6d4dc2abd40828c327677f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:35:31.625715 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8831cb22bf6d4dc2abd40828c327677f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8831cb22bf6d4dc2abd40828c327677f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8831cb22bf6d4dc2abd40828c327677f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45035 } } }
I20260620 23:35:31.626200 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8831cb22bf6d4dc2abd40828c327677f [sys.catalog]: This master's current role is: LEADER
I20260620 23:35:31.627101 1011 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:35:31.627046 1006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8831cb22bf6d4dc2abd40828c327677f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8831cb22bf6d4dc2abd40828c327677f. Latest consensus state: current_term: 1 leader_uuid: "8831cb22bf6d4dc2abd40828c327677f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8831cb22bf6d4dc2abd40828c327677f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45035 } } }
I20260620 23:35:31.627399 1006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8831cb22bf6d4dc2abd40828c327677f [sys.catalog]: This master's current role is: LEADER
I20260620 23:35:31.633473 1011 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:35:31.643765 1011 catalog_manager.cc:1347] Generated new cluster ID: 8f6979ffc9174b6e86bada8ec6fa3f2b
I20260620 23:35:31.643919 1011 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:35:31.655134 1011 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:35:31.657150 1011 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:35:31.665613 1011 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8831cb22bf6d4dc2abd40828c327677f: Generated new TSK 0
I20260620 23:35:31.666508 1011 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 23:35:31.720736 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45035
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:35:32.064916 1025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:35:32.065210 1025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:35:32.065306 1025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:35:32.072412 1025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:35:32.072520 1025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:35:32.072633 1025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 23:35:32.082159 1025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45035
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:35:32.083425 1025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:35:32.084702 1025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:35:32.097779 1030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:35:32.097743 1031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:35:32.097967 1033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:35:32.099080 1025 server_base.cc:1034] running on GCE node
I20260620 23:35:32.099716 1025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:35:32.100749 1025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:35:32.102015 1025 hybrid_clock.cc:648] HybridClock initialized: now 1781998532101965 us; error 44 us; skew 500 ppm
I20260620 23:35:32.102350 1025 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:35:32 dist-test-slave-tcgx krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998532, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:35:32.105677 1025 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 23:35:32.107257 1025 webserver.cc:460] Webserver started at http://127.0.97.1:39127/ using document root <none> and password file <none>
I20260620 23:35:32.107980 1025 fs_manager.cc:359] Metadata directory not provided
I20260620 23:35:32.108094 1025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:35:32.108438 1025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:35:32.111527 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/minicluster-data/ts-0/data/instance:
uuid: "b7b4bf9246e84b3d81d7bb9de4ac927c"
format_stamp: "Formatted at 2026-06-20 23:35:32 on dist-test-slave-tcgx"
server_key: "8d2b1706cbce65b0b6a88ebb31dc5db8"
server_key_iv: "481a35d57307e02b45d2941d4edf29f6"
server_key_version: "encryptionkey@0"
I20260620 23:35:32.112318 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/minicluster-data/ts-0/wal/instance:
uuid: "b7b4bf9246e84b3d81d7bb9de4ac927c"
format_stamp: "Formatted at 2026-06-20 23:35:32 on dist-test-slave-tcgx"
server_key: "8d2b1706cbce65b0b6a88ebb31dc5db8"
server_key_iv: "481a35d57307e02b45d2941d4edf29f6"
server_key_version: "encryptionkey@0"
I20260620 23:35:32.118150 1025 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:35:32.121903 1040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:35:32.123597 1025 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:35:32.123775 1025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/minicluster-data/ts-0/wal
uuid: "b7b4bf9246e84b3d81d7bb9de4ac927c"
format_stamp: "Formatted at 2026-06-20 23:35:32 on dist-test-slave-tcgx"
server_key: "8d2b1706cbce65b0b6a88ebb31dc5db8"
server_key_iv: "481a35d57307e02b45d2941d4edf29f6"
server_key_version: "encryptionkey@0"
I20260620 23:35:32.124027 1025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:35:32.137796 1025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:35:32.142246 1025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:35:32.142539 1025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:35:32.143623 1025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:35:32.144963 1025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:35:32.145044 1025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:35:32.145200 1025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:35:32.145269 1025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:35:32.178064 1025 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46107
I20260620 23:35:32.178099 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46107 every 8 connection(s)
I20260620 23:35:32.179445 1025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
I20260620 23:35:32.180539 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 1025
I20260620 23:35:32.180737 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/minicluster-data/ts-0/wal/instance
I20260620 23:35:32.181615 388 external_mini_cluster.cc:1428] Setting key a7013d2ce1e44f9a9c82a4911bf67792
Jun 20 23:35:32 dist-test-slave-tcgx krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998532, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 23:35:32.203780 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:32.184255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45443 (local address 127.0.97.62:45035)
0620 23:35:32.184795 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:32.184803 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:32.186046 (+ 1243us) server_negotiation.cc:407] Connection header received
0620 23:35:32.188207 (+ 2161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:32.188214 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:32.188327 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:32.188466 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:35:32.190112 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:32.191412 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:32.193071 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:32.193833 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:32.193989 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:32.198183 (+ 4194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:35:32.198190 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:35:32.198195 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:35:32.198243 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:35:32.200908 (+ 2665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:32.201686 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:32.201689 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:32.201692 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:32.201802 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:32.202225 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:32.202228 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:32.202230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:32.202478 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:35:32.202588 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:32.203351 (+ 763us) server_negotiation.cc:299] Negotiation successful
0620 23:35:32.203521 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":277,"threads_started":1}
I20260620 23:35:32.204830 1156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:32.184954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45035 (local address 127.0.97.1:45443)
0620 23:35:32.185835 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0620 23:35:32.185895 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 23:35:32.187898 (+ 2003us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:35:32.188733 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:35:32.188748 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:35:32.189336 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:35:32.189926 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:32.189943 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:32.191560 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:32.191564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:32.192906 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:32.192917 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:32.194048 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:32.194053 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:32.194320 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:32.195173 (+ 853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:35:32.195196 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:35:32.197863 (+ 2667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:35:32.201052 (+ 3189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:35:32.201060 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:35:32.201095 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:35:32.201581 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:35:32.201931 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:35:32.201935 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:35:32.201937 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:35:32.202133 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:35:32.202584 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:35:32.202589 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:35:32.203016 (+ 427us) client_negotiation.cc:769] Sending connection context
0620 23:35:32.203258 (+ 242us) client_negotiation.cc:240] Negotiation successful
0620 23:35:32.203700 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":366,"threads_started":1}
I20260620 23:35:32.206506 1153 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45035
I20260620 23:35:32.206843 1153 heartbeater.cc:464] Registering TS with master...
I20260620 23:35:32.207707 1153 heartbeater.cc:511] Master 127.0.97.62:45035 requested a full tablet report, sending...
I20260620 23:35:32.209905 964 ts_manager.cc:194] Registered new tserver with Master: b7b4bf9246e84b3d81d7bb9de4ac927c (127.0.97.1:46107)
I20260620 23:35:32.211438 964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:45443
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab.
W20260620 23:35:33.183867 388 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.97.2: real 0.999s user 0.000s sys 0.008s
I20260620 23:35:33.184933 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45035
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:35:33.214692 1153 heartbeater.cc:503] Master 127.0.97.62:45035 was elected leader, sending a full tablet report...
W20260620 23:35:33.537094 1159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:35:33.537438 1159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:35:33.537608 1159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:35:33.545408 1159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:35:33.545578 1159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:35:33.545747 1159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 23:35:33.556789 1159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45035
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:35:33.558411 1159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:35:33.560026 1159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:35:33.574028 1167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:35:33.574095 1164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:35:33.574069 1165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:35:33.574831 1159 server_base.cc:1034] running on GCE node
I20260620 23:35:33.575558 1159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:35:33.576593 1159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:35:33.577819 1159 hybrid_clock.cc:648] HybridClock initialized: now 1781998533577778 us; error 37 us; skew 500 ppm
I20260620 23:35:33.578197 1159 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:35:33 dist-test-slave-tcgx krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:35:33.581414 1159 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 23:35:33.583240 1159 webserver.cc:460] Webserver started at http://127.0.97.2:43393/ using document root <none> and password file <none>
I20260620 23:35:33.584023 1159 fs_manager.cc:359] Metadata directory not provided
I20260620 23:35:33.584170 1159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:35:33.584488 1159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:35:33.587682 1159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/minicluster-data/ts-1/data/instance:
uuid: "6bff9ae99c534fc880dd532caa19e4ae"
format_stamp: "Formatted at 2026-06-20 23:35:33 on dist-test-slave-tcgx"
server_key: "37749eccb4e5674f2f9bd724005f1838"
server_key_iv: "93b738ac8fe9694df8119087cdb3bbfc"
server_key_version: "encryptionkey@0"
I20260620 23:35:33.588526 1159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/minicluster-data/ts-1/wal/instance:
uuid: "6bff9ae99c534fc880dd532caa19e4ae"
format_stamp: "Formatted at 2026-06-20 23:35:33 on dist-test-slave-tcgx"
server_key: "37749eccb4e5674f2f9bd724005f1838"
server_key_iv: "93b738ac8fe9694df8119087cdb3bbfc"
server_key_version: "encryptionkey@0"
I20260620 23:35:33.594834 1159 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260620 23:35:33.598630 1174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:35:33.600237 1159 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 23:35:33.600423 1159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/minicluster-data/ts-1/wal
uuid: "6bff9ae99c534fc880dd532caa19e4ae"
format_stamp: "Formatted at 2026-06-20 23:35:33 on dist-test-slave-tcgx"
server_key: "37749eccb4e5674f2f9bd724005f1838"
server_key_iv: "93b738ac8fe9694df8119087cdb3bbfc"
server_key_version: "encryptionkey@0"
I20260620 23:35:33.600641 1159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:35:33.626667 1159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:35:33.631542 1159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:35:33.631896 1159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:35:33.633184 1159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:35:33.634693 1159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:35:33.634802 1159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:35:33.634914 1159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:35:33.634984 1159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260620 23:35:33.670801 1159 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40361
I20260620 23:35:33.670850 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40361 every 8 connection(s)
I20260620 23:35:33.672178 1159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
I20260620 23:35:33.674197 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 1159
I20260620 23:35:33.674392 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/minicluster-data/ts-1/wal/instance
I20260620 23:35:33.674880 388 external_mini_cluster.cc:1428] Setting key 1d5eb4e69ecf4d6505b1fd0e2a753212
Jun 20 23:35:33 dist-test-slave-tcgx krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 23:35:33.699714 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:33.674916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37395 (local address 127.0.97.62:45035)
0620 23:35:33.675402 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:33.675408 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:33.676567 (+ 1159us) server_negotiation.cc:407] Connection header received
0620 23:35:33.679433 (+ 2866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:33.679437 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:33.679503 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:33.679603 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:35:33.681835 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:33.683031 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:33.684746 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:33.685228 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:33.685374 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:33.691152 (+ 5778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:35:33.691161 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:35:33.691166 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:35:33.691219 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:35:33.695793 (+ 4574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:33.696680 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:33.696688 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:33.696695 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:33.696881 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:33.697340 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:33.697348 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:33.697354 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:33.697856 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:35:33.698066 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:33.698879 (+ 813us) server_negotiation.cc:299] Negotiation successful
0620 23:35:33.699219 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":210,"threads_started":1}
I20260620 23:35:33.700497 1290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:33.675625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45035 (local address 127.0.97.2:37395)
0620 23:35:33.676419 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0620 23:35:33.676458 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 23:35:33.679082 (+ 2624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:35:33.679904 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:35:33.679921 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:35:33.680777 (+ 856us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:35:33.681637 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:33.681666 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:33.683199 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:33.683228 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:33.684568 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:33.684580 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:33.685494 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:33.685501 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:33.685825 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:33.687024 (+ 1199us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:35:33.687063 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:35:33.690739 (+ 3676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:35:33.696018 (+ 5279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:35:33.696027 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:35:33.696042 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:35:33.696509 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:35:33.697016 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:35:33.697020 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:35:33.697022 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:35:33.697222 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:35:33.698045 (+ 823us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:35:33.698055 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:35:33.698445 (+ 390us) client_negotiation.cc:769] Sending connection context
0620 23:35:33.698719 (+ 274us) client_negotiation.cc:240] Negotiation successful
0620 23:35:33.699272 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":328,"threads_started":1}
I20260620 23:35:33.702116 1287 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45035
I20260620 23:35:33.702486 1287 heartbeater.cc:464] Registering TS with master...
I20260620 23:35:33.703501 1287 heartbeater.cc:511] Master 127.0.97.62:45035 requested a full tablet report, sending...
I20260620 23:35:33.705307 964 ts_manager.cc:194] Registered new tserver with Master: 6bff9ae99c534fc880dd532caa19e4ae (127.0.97.2:40361)
I20260620 23:35:33.706205 964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:37395
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab.
W20260620 23:35:34.263152 388 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.97.3: real 0.585s user 0.000s sys 0.011s
I20260620 23:35:34.263929 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45035
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:35:34.601758 1293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:35:34.602088 1293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:35:34.602253 1293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:35:34.609882 1293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:35:34.610001 1293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:35:34.610205 1293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 23:35:34.620131 1293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45035
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:35:34.621505 1293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:35:34.622871 1293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:35:34.636509 1299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:35:34.636490 1298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:35:34.636616 1301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:35:34.637256 1293 server_base.cc:1034] running on GCE node
I20260620 23:35:34.637894 1293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:35:34.638923 1293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:35:34.640203 1293 hybrid_clock.cc:648] HybridClock initialized: now 1781998534640119 us; error 67 us; skew 500 ppm
I20260620 23:35:34.640571 1293 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:35:34 dist-test-slave-tcgx krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998534, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:35:34.643884 1293 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 23:35:34.645515 1293 webserver.cc:460] Webserver started at http://127.0.97.3:37141/ using document root <none> and password file <none>
I20260620 23:35:34.646255 1293 fs_manager.cc:359] Metadata directory not provided
I20260620 23:35:34.646412 1293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:35:34.646744 1293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:35:34.649902 1293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/minicluster-data/ts-2/data/instance:
uuid: "f77f5c3367cd4d2b9ce41a73b2122f0c"
format_stamp: "Formatted at 2026-06-20 23:35:34 on dist-test-slave-tcgx"
server_key: "0108d3f72b91a13fb84aa7cc4f4cf898"
server_key_iv: "b55cc940d2908351014460a48a6083a7"
server_key_version: "encryptionkey@0"
I20260620 23:35:34.650696 1293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/minicluster-data/ts-2/wal/instance:
uuid: "f77f5c3367cd4d2b9ce41a73b2122f0c"
format_stamp: "Formatted at 2026-06-20 23:35:34 on dist-test-slave-tcgx"
server_key: "0108d3f72b91a13fb84aa7cc4f4cf898"
server_key_iv: "b55cc940d2908351014460a48a6083a7"
server_key_version: "encryptionkey@0"
I20260620 23:35:34.656418 1293 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 23:35:34.659978 1308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:35:34.661345 1293 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 23:35:34.661523 1293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/minicluster-data/ts-2/wal
uuid: "f77f5c3367cd4d2b9ce41a73b2122f0c"
format_stamp: "Formatted at 2026-06-20 23:35:34 on dist-test-slave-tcgx"
server_key: "0108d3f72b91a13fb84aa7cc4f4cf898"
server_key_iv: "b55cc940d2908351014460a48a6083a7"
server_key_version: "encryptionkey@0"
I20260620 23:35:34.661732 1293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:35:34.682176 1293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:35:34.686990 1293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:35:34.687304 1293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:35:34.688330 1293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:35:34.689705 1293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:35:34.689816 1293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:35:34.689971 1293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:35:34.690044 1293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:35:34.709056 1287 heartbeater.cc:503] Master 127.0.97.62:45035 was elected leader, sending a full tablet report...
I20260620 23:35:34.725071 1293 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33093
I20260620 23:35:34.725114 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33093 every 8 connection(s)
I20260620 23:35:34.726384 1293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
I20260620 23:35:34.731354 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 1293
I20260620 23:35:34.731529 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998525501163-388-0/minicluster-data/ts-2/wal/instance
I20260620 23:35:34.732084 388 external_mini_cluster.cc:1428] Setting key 2b22f9dd01bb8b1592608de66566d2b2
Jun 20 23:35:34 dist-test-slave-tcgx krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998534, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:35:34.748685 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:34.729293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57617 (local address 127.0.97.62:45035)
0620 23:35:34.729811 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:34.729816 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:34.730747 (+ 931us) server_negotiation.cc:407] Connection header received
0620 23:35:34.733238 (+ 2491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:34.733243 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:34.733335 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:34.733470 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:35:34.735451 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:34.736223 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:34.737768 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:34.738180 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:34.738280 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:34.742395 (+ 4115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:35:34.742401 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:35:34.742405 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:35:34.742454 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:35:34.745152 (+ 2698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:34.745975 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:34.745981 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:34.745984 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:34.746144 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:34.746622 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:34.746627 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:34.746629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:34.747132 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:35:34.747310 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:34.748222 (+ 912us) server_negotiation.cc:299] Negotiation successful
0620 23:35:34.748410 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":284,"threads_started":1}
I20260620 23:35:34.749823 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:34.729833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45035 (local address 127.0.97.3:57617)
0620 23:35:34.730562 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0620 23:35:34.730609 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 23:35:34.732950 (+ 2341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:35:34.733741 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:35:34.733754 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:35:34.734484 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:35:34.735288 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:34.735307 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:34.736357 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:34.736362 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:34.737626 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:34.737637 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:34.738378 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:34.738383 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:34.739070 (+ 687us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:34.739795 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:35:34.739821 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:35:34.742164 (+ 2343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:35:34.745310 (+ 3146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:35:34.745318 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:35:34.745333 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:35:34.745812 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:35:34.746265 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:35:34.746270 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:35:34.746273 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:35:34.746474 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:35:34.747286 (+ 812us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:35:34.747292 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:35:34.747738 (+ 446us) client_negotiation.cc:769] Sending connection context
0620 23:35:34.748129 (+ 391us) client_negotiation.cc:240] Negotiation successful
0620 23:35:34.748591 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":279,"threads_started":1}
I20260620 23:35:34.751335 1421 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45035
I20260620 23:35:34.751650 1421 heartbeater.cc:464] Registering TS with master...
I20260620 23:35:34.752404 1421 heartbeater.cc:511] Master 127.0.97.62:45035 requested a full tablet report, sending...
I20260620 23:35:34.753804 964 ts_manager.cc:194] Registered new tserver with Master: f77f5c3367cd4d2b9ce41a73b2122f0c (127.0.97.3:33093)
I20260620 23:35:34.754586 964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:57617
I20260620 23:35:34.760620 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:35:34.780015 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:34.767285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51482 (local address 127.0.97.62:45035)
0620 23:35:34.767488 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:34.767508 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:34.768014 (+ 506us) server_negotiation.cc:407] Connection header received
0620 23:35:34.768159 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:34.768163 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:34.768241 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:34.768333 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:35:34.770306 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:34.771125 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:34.772378 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:34.772779 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:34.772875 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:34.775234 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:35:34.775240 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:35:34.775243 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:35:34.775278 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:35:34.777656 (+ 2378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:34.778314 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:34.778318 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:34.778320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:34.778437 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:34.778740 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:34.778743 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:34.778745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:34.779108 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:35:34.779286 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:34.779600 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 23:35:34.779763 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 23:35:34.786655 964 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51482:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:35:34.793951 964 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:35:34.829284 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:34.807933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55526 (local address 127.0.97.1:46107)
0620 23:35:34.818364 (+ 10431us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:34.818378 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:34.821187 (+ 2809us) server_negotiation.cc:407] Connection header received
0620 23:35:34.821293 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:34.821300 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:34.821545 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:34.821745 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:35:34.822290 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:34.823777 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:34.826396 (+ 2619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:34.827904 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:34.828045 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:34.828282 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:34.828416 (+ 134us) server_negotiation.cc:299] Negotiation successful
0620 23:35:34.828576 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10184,"thread_start_us":9984,"threads_started":1}
I20260620 23:35:34.829435 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:34.806191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40361 (local address 127.0.0.1:41312)
0620 23:35:34.807102 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0620 23:35:34.807146 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 23:35:34.807303 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:35:34.820320 (+ 13017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:35:34.820334 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:35:34.820365 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:35:34.820946 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:34.820957 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:34.823045 (+ 2088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:34.823051 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:34.825690 (+ 2639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:34.825705 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:34.827695 (+ 1990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:34.827702 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:34.827923 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:34.828134 (+ 211us) client_negotiation.cc:769] Sending connection context
0620 23:35:34.828269 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 23:35:34.828503 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"mutex_wait_us":90,"thread_start_us":466,"threads_started":1}
I20260620 23:35:34.829656 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:34.807540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46107 (local address 127.0.0.1:55526)
0620 23:35:34.821035 (+ 13495us) negotiation.cc:107] Waiting for socket to connect
0620 23:35:34.821053 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:35:34.821174 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:35:34.821746 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:35:34.821752 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:35:34.821784 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:35:34.822124 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:34.822135 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:34.823991 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:34.823998 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:34.826233 (+ 2235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:34.826246 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:34.828146 (+ 1900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:34.828152 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:34.828264 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:34.828287 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 23:35:34.828357 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 23:35:34.828532 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13377,"mutex_wait_us":26,"thread_start_us":13205,"threads_started":1}
I20260620 23:35:34.830719 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:34.806297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56368 (local address 127.0.97.3:33093)
0620 23:35:34.818838 (+ 12541us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:34.818850 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:34.820488 (+ 1638us) server_negotiation.cc:407] Connection header received
0620 23:35:34.820828 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:34.820837 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:34.821123 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:34.821288 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:35:34.821954 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:34.823182 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:34.826211 (+ 3029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:34.827269 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:34.827423 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:34.827612 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:34.830210 (+ 2598us) server_negotiation.cc:299] Negotiation successful
0620 23:35:34.830396 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12312,"thread_start_us":12140,"threads_started":1}
I20260620 23:35:34.832448 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:34.806898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41312 (local address 127.0.97.2:40361)
0620 23:35:34.819512 (+ 12614us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:34.819521 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:34.819554 (+ 33us) server_negotiation.cc:407] Connection header received
0620 23:35:34.819664 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:34.819673 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:34.820024 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:34.821169 (+ 1145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:35:34.821388 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:34.822853 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:34.825836 (+ 2983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:34.827404 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:34.831579 (+ 4175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:34.831885 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:34.831962 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 23:35:34.832145 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12430,"thread_start_us":12209,"threads_started":1}
I20260620 23:35:34.832635 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:34.806231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33093 (local address 127.0.0.1:56368)
0620 23:35:34.820347 (+ 14116us) negotiation.cc:107] Waiting for socket to connect
0620 23:35:34.820376 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 23:35:34.820607 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:35:34.821339 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:35:34.821346 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:35:34.821370 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:35:34.821710 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:34.821722 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:34.823320 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:34.823325 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:34.826042 (+ 2717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:34.826056 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:34.827733 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:34.827737 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:34.828050 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:34.830032 (+ 1982us) client_negotiation.cc:769] Sending connection context
0620 23:35:34.830130 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 23:35:34.830265 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13987,"mutex_wait_us":2303,"thread_start_us":13793,"threads_started":1}
I20260620 23:35:34.834141 1088 tablet_service.cc:1467] Processing CreateTablet for tablet 36f5e5b73ae447e897d3b7525bf3a7a6 (DEFAULT_TABLE table=test-table [id=dad4559917aa4a24a65a1e6717f3515e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:35:34.835741 1088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36f5e5b73ae447e897d3b7525bf3a7a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:35:34.836130 1356 tablet_service.cc:1467] Processing CreateTablet for tablet 36f5e5b73ae447e897d3b7525bf3a7a6 (DEFAULT_TABLE table=test-table [id=dad4559917aa4a24a65a1e6717f3515e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:35:34.836146 1222 tablet_service.cc:1467] Processing CreateTablet for tablet 36f5e5b73ae447e897d3b7525bf3a7a6 (DEFAULT_TABLE table=test-table [id=dad4559917aa4a24a65a1e6717f3515e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:35:34.837353 1356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36f5e5b73ae447e897d3b7525bf3a7a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:35:34.837522 1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36f5e5b73ae447e897d3b7525bf3a7a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:35:34.849202 1440 tablet_bootstrap.cc:492] T 36f5e5b73ae447e897d3b7525bf3a7a6 P f77f5c3367cd4d2b9ce41a73b2122f0c: Bootstrap starting.
I20260620 23:35:34.851292 1441 tablet_bootstrap.cc:492] T 36f5e5b73ae447e897d3b7525bf3a7a6 P b7b4bf9246e84b3d81d7bb9de4ac927c: Bootstrap starting.
I20260620 23:35:34.852820 1442 tablet_bootstrap.cc:492] T 36f5e5b73ae447e897d3b7525bf3a7a6 P 6bff9ae99c534fc880dd532caa19e4ae: Bootstrap starting.
I20260620 23:35:34.854364 1440 tablet_bootstrap.cc:654] T 36f5e5b73ae447e897d3b7525bf3a7a6 P f77f5c3367cd4d2b9ce41a73b2122f0c: Neither blocks nor log segments found. Creating new log.
I20260620 23:35:34.854902 1441 tablet_bootstrap.cc:654] T 36f5e5b73ae447e897d3b7525bf3a7a6 P b7b4bf9246e84b3d81d7bb9de4ac927c: Neither blocks nor log segments found. Creating new log.
I20260620 23:35:34.855700 1442 tablet_bootstrap.cc:654] T 36f5e5b73ae447e897d3b7525bf3a7a6 P 6bff9ae99c534fc880dd532caa19e4ae: Neither blocks nor log segments found. Creating new log.
I20260620 23:35:34.856237 1440 log.cc:824] T 36f5e5b73ae447e897d3b7525bf3a7a6 P f77f5c3367cd4d2b9ce41a73b2122f0c: Log is configured to *not* fsync() on all Append() calls
I20260620 23:35:34.856590 1441 log.cc:824] T 36f5e5b73ae447e897d3b7525bf3a7a6 P b7b4bf9246e84b3d81d7bb9de4ac927c: Log is configured to *not* fsync() on all Append() calls
I20260620 23:35:34.857203 1442 log.cc:824] T 36f5e5b73ae447e897d3b7525bf3a7a6 P 6bff9ae99c534fc880dd532caa19e4ae: Log is configured to *not* fsync() on all Append() calls
I20260620 23:35:34.859978 1440 tablet_bootstrap.cc:492] T 36f5e5b73ae447e897d3b7525bf3a7a6 P f77f5c3367cd4d2b9ce41a73b2122f0c: No bootstrap required, opened a new log
I20260620 23:35:34.860088 1442 tablet_bootstrap.cc:492] T 36f5e5b73ae447e897d3b7525bf3a7a6 P 6bff9ae99c534fc880dd532caa19e4ae: No bootstrap required, opened a new log
I20260620 23:35:34.860282 1440 ts_tablet_manager.cc:1397] T 36f5e5b73ae447e897d3b7525bf3a7a6 P f77f5c3367cd4d2b9ce41a73b2122f0c: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260620 23:35:34.860368 1442 ts_tablet_manager.cc:1397] T 36f5e5b73ae447e897d3b7525bf3a7a6 P 6bff9ae99c534fc880dd532caa19e4ae: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 23:35:34.867642 1441 tablet_bootstrap.cc:492] T 36f5e5b73ae447e897d3b7525bf3a7a6 P b7b4bf9246e84b3d81d7bb9de4ac927c: No bootstrap required, opened a new log
I20260620 23:35:34.867996 1441 ts_tablet_manager.cc:1397] T 36f5e5b73ae447e897d3b7525bf3a7a6 P b7b4bf9246e84b3d81d7bb9de4ac927c: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.000s
I20260620 23:35:34.868220 1440 raft_consensus.cc:348] T 36f5e5b73ae447e897d3b7525bf3a7a6 P f77f5c3367cd4d2b9ce41a73b2122f0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f77f5c3367cd4d2b9ce41a73b2122f0c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33093 } } peers { permanent_uuid: "6bff9ae99c534fc880dd532caa19e4ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40361 } } peers { permanent_uuid: "b7b4bf9246e84b3d81d7bb9de4ac927c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46107 } }
I20260620 23:35:34.868606 1440 raft_consensus.cc:374] T 36f5e5b73ae447e897d3b7525bf3a7a6 P f77f5c3367cd4d2b9ce41a73b2122f0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:35:34.868698 1440 raft_consensus.cc:729] T 36f5e5b73ae447e897d3b7525bf3a7a6 P f77f5c3367cd4d2b9ce41a73b2122f0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f77f5c3367cd4d2b9ce41a73b2122f0c, State: Initialized, Role: FOLLOWER
I20260620 23:35:34.869338 1440 consensus_queue.cc:260] T 36f5e5b73ae447e897d3b7525bf3a7a6 P f77f5c3367cd4d2b9ce41a73b2122f0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f77f5c3367cd4d2b9ce41a73b2122f0c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33093 } } peers { permanent_uuid: "6bff9ae99c534fc880dd532caa19e4ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40361 } } peers { permanent_uuid: "b7b4bf9246e84b3d81d7bb9de4ac927c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46107 } }
I20260620 23:35:34.869251 1442 raft_consensus.cc:348] T 36f5e5b73ae447e897d3b7525bf3a7a6 P 6bff9ae99c534fc880dd532caa19e4ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f77f5c3367cd4d2b9ce41a73b2122f0c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33093 } } peers { permanent_uuid: "6bff9ae99c534fc880dd532caa19e4ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40361 } } peers { permanent_uuid: "b7b4bf9246e84b3d81d7bb9de4ac927c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46107 } }
I20260620 23:35:34.869683 1442 raft_consensus.cc:374] T 36f5e5b73ae447e897d3b7525bf3a7a6 P 6bff9ae99c534fc880dd532caa19e4ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:35:34.869786 1442 raft_consensus.cc:729] T 36f5e5b73ae447e897d3b7525bf3a7a6 P 6bff9ae99c534fc880dd532caa19e4ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bff9ae99c534fc880dd532caa19e4ae, State: Initialized, Role: FOLLOWER
I20260620 23:35:34.870545 1421 heartbeater.cc:503] Master 127.0.97.62:45035 was elected leader, sending a full tablet report...
I20260620 23:35:34.870483 1442 consensus_queue.cc:260] T 36f5e5b73ae447e897d3b7525bf3a7a6 P 6bff9ae99c534fc880dd532caa19e4ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f77f5c3367cd4d2b9ce41a73b2122f0c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33093 } } peers { permanent_uuid: "6bff9ae99c534fc880dd532caa19e4ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40361 } } peers { permanent_uuid: "b7b4bf9246e84b3d81d7bb9de4ac927c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46107 } }
I20260620 23:35:34.871279 1440 ts_tablet_manager.cc:1428] T 36f5e5b73ae447e897d3b7525bf3a7a6 P f77f5c3367cd4d2b9ce41a73b2122f0c: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260620 23:35:34.875077 1442 ts_tablet_manager.cc:1428] T 36f5e5b73ae447e897d3b7525bf3a7a6 P 6bff9ae99c534fc880dd532caa19e4ae: Time spent starting tablet: real 0.015s user 0.011s sys 0.003s
I20260620 23:35:34.878747 1441 raft_consensus.cc:348] T 36f5e5b73ae447e897d3b7525bf3a7a6 P b7b4bf9246e84b3d81d7bb9de4ac927c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f77f5c3367cd4d2b9ce41a73b2122f0c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33093 } } peers { permanent_uuid: "6bff9ae99c534fc880dd532caa19e4ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40361 } } peers { permanent_uuid: "b7b4bf9246e84b3d81d7bb9de4ac927c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46107 } }
I20260620 23:35:34.879246 1441 raft_consensus.cc:374] T 36f5e5b73ae447e897d3b7525bf3a7a6 P b7b4bf9246e84b3d81d7bb9de4ac927c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:35:34.879366 1441 raft_consensus.cc:729] T 36f5e5b73ae447e897d3b7525bf3a7a6 P b7b4bf9246e84b3d81d7bb9de4ac927c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7b4bf9246e84b3d81d7bb9de4ac927c, State: Initialized, Role: FOLLOWER
I20260620 23:35:34.879999 1441 consensus_queue.cc:260] T 36f5e5b73ae447e897d3b7525bf3a7a6 P b7b4bf9246e84b3d81d7bb9de4ac927c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f77f5c3367cd4d2b9ce41a73b2122f0c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33093 } } peers { permanent_uuid: "6bff9ae99c534fc880dd532caa19e4ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40361 } } peers { permanent_uuid: "b7b4bf9246e84b3d81d7bb9de4ac927c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46107 } }
I20260620 23:35:34.881767 1441 ts_tablet_manager.cc:1428] T 36f5e5b73ae447e897d3b7525bf3a7a6 P b7b4bf9246e84b3d81d7bb9de4ac927c: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260620 23:35:34.909015 1447 raft_consensus.cc:482] T 36f5e5b73ae447e897d3b7525bf3a7a6 P 6bff9ae99c534fc880dd532caa19e4ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:35:34.909358 1447 raft_consensus.cc:504] T 36f5e5b73ae447e897d3b7525bf3a7a6 P 6bff9ae99c534fc880dd532caa19e4ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f77f5c3367cd4d2b9ce41a73b2122f0c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33093 } } peers { permanent_uuid: "6bff9ae99c534fc880dd532caa19e4ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40361 } } peers { permanent_uuid: "b7b4bf9246e84b3d81d7bb9de4ac927c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46107 } }
I20260620 23:35:34.911121 1447 leader_election.cc:288] T 36f5e5b73ae447e897d3b7525bf3a7a6 P 6bff9ae99c534fc880dd532caa19e4ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f77f5c3367cd4d2b9ce41a73b2122f0c (127.0.97.3:33093), b7b4bf9246e84b3d81d7bb9de4ac927c (127.0.97.1:46107)
I20260620 23:35:34.918879 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:34.911653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51721 (local address 127.0.97.1:46107)
0620 23:35:34.911900 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:34.911909 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:34.913451 (+ 1542us) server_negotiation.cc:407] Connection header received
0620 23:35:34.913648 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:34.913652 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:34.913722 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:34.913815 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:35:34.914470 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:34.915315 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:34.917186 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:34.917935 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:34.918094 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:34.918201 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:34.918481 (+ 280us) server_negotiation.cc:299] Negotiation successful
0620 23:35:34.918609 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 23:35:34.919128 1450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:34.911718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46107 (local address 127.0.97.2:51721)
0620 23:35:34.913310 (+ 1592us) negotiation.cc:107] Waiting for socket to connect
0620 23:35:34.913330 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:35:34.913506 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:35:34.913844 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:35:34.913848 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:35:34.913872 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:35:34.914282 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:34.914291 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:34.915455 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:34.915462 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:34.917030 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:34.917042 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:34.918151 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:34.918157 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:34.918321 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:34.918345 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 23:35:34.918456 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 23:35:34.918593 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1495,"mutex_wait_us":262,"thread_start_us":1086,"threads_started":1}
I20260620 23:35:34.919586 1449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:34.911712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33093 (local address 127.0.97.2:34799)
0620 23:35:34.913171 (+ 1459us) negotiation.cc:107] Waiting for socket to connect
0620 23:35:34.913199 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 23:35:34.913417 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:35:34.913803 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:35:34.913808 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:35:34.913835 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:35:34.914281 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:34.914291 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:34.915563 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:34.915569 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:34.917361 (+ 1792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:34.917376 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:34.918628 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:34.918634 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:34.918807 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:34.918831 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 23:35:34.918972 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 23:35:34.919110 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1270,"thread_start_us":427,"threads_started":1}
I20260620 23:35:34.919780 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:34.911686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34799 (local address 127.0.97.3:33093)
0620 23:35:34.911871 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:34.911877 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:34.913332 (+ 1455us) server_negotiation.cc:407] Connection header received
0620 23:35:34.913597 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:34.913602 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:34.913669 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:34.913774 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:35:34.914449 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:34.915419 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:34.917533 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:34.918390 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:34.918556 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:34.918672 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:34.918995 (+ 323us) server_negotiation.cc:299] Negotiation successful
0620 23:35:34.919507 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 23:35:34.920214 1108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f5e5b73ae447e897d3b7525bf3a7a6" candidate_uuid: "6bff9ae99c534fc880dd532caa19e4ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7b4bf9246e84b3d81d7bb9de4ac927c" is_pre_election: true
I20260620 23:35:34.920747 1108 raft_consensus.cc:2454] T 36f5e5b73ae447e897d3b7525bf3a7a6 P b7b4bf9246e84b3d81d7bb9de4ac927c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bff9ae99c534fc880dd532caa19e4ae in term 0.
I20260620 23:35:34.920696 1376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f5e5b73ae447e897d3b7525bf3a7a6" candidate_uuid: "6bff9ae99c534fc880dd532caa19e4ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f77f5c3367cd4d2b9ce41a73b2122f0c" is_pre_election: true
I20260620 23:35:34.921180 1376 raft_consensus.cc:2454] T 36f5e5b73ae447e897d3b7525bf3a7a6 P f77f5c3367cd4d2b9ce41a73b2122f0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bff9ae99c534fc880dd532caa19e4ae in term 0.
I20260620 23:35:34.921463 1178 leader_election.cc:302] T 36f5e5b73ae447e897d3b7525bf3a7a6 P 6bff9ae99c534fc880dd532caa19e4ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bff9ae99c534fc880dd532caa19e4ae, b7b4bf9246e84b3d81d7bb9de4ac927c; no voters:
I20260620 23:35:34.921864 1447 raft_consensus.cc:2781] T 36f5e5b73ae447e897d3b7525bf3a7a6 P 6bff9ae99c534fc880dd532caa19e4ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:35:34.922021 1447 raft_consensus.cc:482] T 36f5e5b73ae447e897d3b7525bf3a7a6 P 6bff9ae99c534fc880dd532caa19e4ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:35:34.922138 1447 raft_consensus.cc:3037] T 36f5e5b73ae447e897d3b7525bf3a7a6 P 6bff9ae99c534fc880dd532caa19e4ae [term 0 FOLLOWER]: Advancing to term 1
W20260620 23:35:34.924515 1288 tablet.cc:2369] T 36f5e5b73ae447e897d3b7525bf3a7a6 P 6bff9ae99c534fc880dd532caa19e4ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:35:34.925287 1447 raft_consensus.cc:504] T 36f5e5b73ae447e897d3b7525bf3a7a6 P 6bff9ae99c534fc880dd532caa19e4ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f77f5c3367cd4d2b9ce41a73b2122f0c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33093 } } peers { permanent_uuid: "6bff9ae99c534fc880dd532caa19e4ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40361 } } peers { permanent_uuid: "b7b4bf9246e84b3d81d7bb9de4ac927c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46107 } }
I20260620 23:35:34.925910 1447 leader_election.cc:288] T 36f5e5b73ae447e897d3b7525bf3a7a6 P 6bff9ae99c534fc880dd532caa19e4ae [CANDIDATE]: Term 1 election: Requested vote from peers f77f5c3367cd4d2b9ce41a73b2122f0c (127.0.97.3:33093), b7b4bf9246e84b3d81d7bb9de4ac927c (127.0.97.1:46107)
I20260620 23:35:34.926373 1376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f5e5b73ae447e897d3b7525bf3a7a6" candidate_uuid: "6bff9ae99c534fc880dd532caa19e4ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f77f5c3367cd4d2b9ce41a73b2122f0c"
I20260620 23:35:34.926390 1108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36f5e5b73ae447e897d3b7525bf3a7a6" candidate_uuid: "6bff9ae99c534fc880dd532caa19e4ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7b4bf9246e84b3d81d7bb9de4ac927c"
I20260620 23:35:34.926585 1376 raft_consensus.cc:3037] T 36f5e5b73ae447e897d3b7525bf3a7a6 P f77f5c3367cd4d2b9ce41a73b2122f0c [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:35:34.926599 1108 raft_consensus.cc:3037] T 36f5e5b73ae447e897d3b7525bf3a7a6 P b7b4bf9246e84b3d81d7bb9de4ac927c [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:35:34.929458 1108 raft_consensus.cc:2454] T 36f5e5b73ae447e897d3b7525bf3a7a6 P b7b4bf9246e84b3d81d7bb9de4ac927c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bff9ae99c534fc880dd532caa19e4ae in term 1.
I20260620 23:35:34.929513 1376 raft_consensus.cc:2454] T 36f5e5b73ae447e897d3b7525bf3a7a6 P f77f5c3367cd4d2b9ce41a73b2122f0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bff9ae99c534fc880dd532caa19e4ae in term 1.
I20260620 23:35:34.929971 1178 leader_election.cc:302] T 36f5e5b73ae447e897d3b7525bf3a7a6 P 6bff9ae99c534fc880dd532caa19e4ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bff9ae99c534fc880dd532caa19e4ae, b7b4bf9246e84b3d81d7bb9de4ac927c; no voters:
I20260620 23:35:34.930310 1447 raft_consensus.cc:2781] T 36f5e5b73ae447e897d3b7525bf3a7a6 P 6bff9ae99c534fc880dd532caa19e4ae [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:35:34.930765 1447 raft_consensus.cc:686] T 36f5e5b73ae447e897d3b7525bf3a7a6 P 6bff9ae99c534fc880dd532caa19e4ae [term 1 LEADER]: Becoming Leader. State: Replica: 6bff9ae99c534fc880dd532caa19e4ae, State: Running, Role: LEADER
I20260620 23:35:34.931339 1447 consensus_queue.cc:237] T 36f5e5b73ae447e897d3b7525bf3a7a6 P 6bff9ae99c534fc880dd532caa19e4ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f77f5c3367cd4d2b9ce41a73b2122f0c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33093 } } peers { permanent_uuid: "6bff9ae99c534fc880dd532caa19e4ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40361 } } peers { permanent_uuid: "b7b4bf9246e84b3d81d7bb9de4ac927c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46107 } }
W20260620 23:35:34.934634 1154 tablet.cc:2369] T 36f5e5b73ae447e897d3b7525bf3a7a6 P b7b4bf9246e84b3d81d7bb9de4ac927c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:35:34.935987 963 catalog_manager.cc:5495] T 36f5e5b73ae447e897d3b7525bf3a7a6 P 6bff9ae99c534fc880dd532caa19e4ae reported cstate change: term changed from 0 to 1, leader changed from <none> to 6bff9ae99c534fc880dd532caa19e4ae (127.0.97.2). New cstate: current_term: 1 leader_uuid: "6bff9ae99c534fc880dd532caa19e4ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f77f5c3367cd4d2b9ce41a73b2122f0c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6bff9ae99c534fc880dd532caa19e4ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b7b4bf9246e84b3d81d7bb9de4ac927c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46107 } health_report { overall_health: UNKNOWN } } }
Jun 20 23:35:34 dist-test-slave-tcgx krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
W20260620 23:35:34.978209 1422 tablet.cc:2369] T 36f5e5b73ae447e897d3b7525bf3a7a6 P f77f5c3367cd4d2b9ce41a73b2122f0c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:35:34.980291 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:34.965175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55530 (local address 127.0.97.1:46107)
0620 23:35:34.965440 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:34.965445 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:34.966152 (+ 707us) server_negotiation.cc:407] Connection header received
0620 23:35:34.966366 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:34.966371 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:34.966478 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:34.966594 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:35:34.968091 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:34.968822 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:34.970074 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:34.970565 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:34.970700 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:34.974567 (+ 3867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:35:34.974583 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:35:34.974599 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:35:34.974640 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:35:34.977407 (+ 2767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:34.978145 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:34.978153 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:34.978155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:34.978286 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:34.978694 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:34.978700 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:34.978704 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:34.979140 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:35:34.979436 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:34.979735 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 23:35:34.979946 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
Jun 20 23:35:35 dist-test-slave-tcgx krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998535, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 23:35:35 dist-test-slave-tcgx krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998535, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 23:35:35.027259 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:35.012615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55534 (local address 127.0.97.1:46107)
0620 23:35:35.012875 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:35.012883 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:35.013217 (+ 334us) server_negotiation.cc:407] Connection header received
0620 23:35:35.013341 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:35.013346 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:35.013430 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:35.013531 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:35:35.014849 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:35.015854 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:35.017220 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:35.017685 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:35.017796 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:35.021763 (+ 3967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:35:35.021768 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:35:35.021772 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:35:35.021811 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:35:35.024727 (+ 2916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:35.025469 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:35.025473 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:35.025476 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:35.025596 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:35.025944 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:35.025948 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:35.025951 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:35.026261 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:35:35.026482 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:35.026714 (+ 232us) server_negotiation.cc:299] Negotiation successful
0620 23:35:35.027000 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
Jun 20 23:35:35 dist-test-slave-tcgx krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 23:35:35 dist-test-slave-tcgx krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 23:35:35.072664 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:35.057938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55540 (local address 127.0.97.1:46107)
0620 23:35:35.058185 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:35.058192 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:35.058508 (+ 316us) server_negotiation.cc:407] Connection header received
0620 23:35:35.058736 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:35.058740 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:35.058807 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:35.058915 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:35:35.060720 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:35.061414 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:35.062772 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:35.063263 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:35.063400 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:35.067484 (+ 4084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:35:35.067489 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:35:35.067493 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:35:35.067532 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:35:35.069926 (+ 2394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:35.070614 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:35.070622 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:35.070627 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:35.070758 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:35.071187 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:35.071193 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:35.071197 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:35.071501 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:35:35.071756 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:35.072065 (+ 309us) server_negotiation.cc:299] Negotiation successful
0620 23:35:35.072313 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260620 23:35:35.074420 1068 tablet_service.cc:2908] Rejecting scan request for tablet 36f5e5b73ae447e897d3b7525bf3a7a6: Uninitialized: clean time has not yet been initialized
I20260620 23:35:35.077476 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 1025
I20260620 23:35:35.091068 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:35.078401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55550 (local address 127.0.97.1:46107)
0620 23:35:35.078556 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:35.078560 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:35.079012 (+ 452us) server_negotiation.cc:407] Connection header received
0620 23:35:35.079181 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:35.079184 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:35.079255 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:35.079336 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:35:35.080545 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:35.081208 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:35.082382 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:35.082769 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:35.082871 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:35.085157 (+ 2286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:35:35.085162 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:35:35.085165 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:35:35.085200 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:35:35.087674 (+ 2474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:35.088659 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:35.088664 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:35.088670 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:35.088806 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:35.089201 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:35.089209 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:35.089212 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:35.089498 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:35:35.089789 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:35.090499 (+ 710us) server_negotiation.cc:299] Negotiation successful
0620 23:35:35.090716 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 23:35:35.091768 1148 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:35:35.182188 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 1025
I20260620 23:35:35.198908 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 1159
Jun 20 23:35:35 dist-test-slave-tcgx krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 23:35:35.213647 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:35.199744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41322 (local address 127.0.97.2:40361)
0620 23:35:35.199939 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:35.199943 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:35.199963 (+ 20us) server_negotiation.cc:407] Connection header received
0620 23:35:35.200105 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:35.200109 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:35.200169 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:35.200251 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:35:35.201388 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:35.202127 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:35.203439 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:35.203827 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:35.203988 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:35.208002 (+ 4014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:35:35.208016 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:35:35.208032 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:35:35.208070 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:35:35.210812 (+ 2742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:35.211726 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:35.211734 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:35.211736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:35.211904 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:35.212260 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:35.212264 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:35.212266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:35.212603 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:35:35.212817 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:35.213111 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 23:35:35.213283 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 23:35:35.214507 1282 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:35:35.306766 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 1159
I20260620 23:35:35.323963 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 1293
Jun 20 23:35:35 dist-test-slave-tcgx krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 23:35:35.339438 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:35.324750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56382 (local address 127.0.97.3:33093)
0620 23:35:35.324939 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:35.324944 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:35.324984 (+ 40us) server_negotiation.cc:407] Connection header received
0620 23:35:35.325151 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:35.325155 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:35.325220 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:35.325315 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:35:35.326493 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:35.327322 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:35.328612 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:35.328987 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:35.329077 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:35.333675 (+ 4598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:35:35.333693 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:35:35.333709 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:35:35.333746 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:35:35.336596 (+ 2850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:35.337364 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:35.337371 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:35.337373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:35.337510 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:35.337836 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:35.337839 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:35.337841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:35.338195 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:35:35.338657 (+ 462us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:35.338858 (+ 201us) server_negotiation.cc:299] Negotiation successful
0620 23:35:35.339186 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 23:35:35.340091 1416 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:35:35.428830 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 1293
I20260620 23:35:35.445052 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 932
I20260620 23:35:35.446252 994 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:35:35.560636 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 932
2026-06-20T23:35:35Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (5664 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:35:35 dist-test-slave-tcgx krb5kdc[1470](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:35:35 dist-test-slave-tcgx krb5kdc[1470](info): set up 2 sockets
Jun 20 23:35:35 dist-test-slave-tcgx krb5kdc[1470](info): commencing operation
krb5kdc: starting...
W20260620 23:35:36.655077 388 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/krb5kdc/test-user.keytab.
Jun 20 23:35:36 dist-test-slave-tcgx krb5kdc[1470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:35:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:35:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 23:35:36.839061 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44817
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44817
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:35:37.160710 1483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:35:37.161029 1483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:35:37.161206 1483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:35:37.168555 1483 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:35:37.168670 1483 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 23:35:37.168764 1483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:35:37.168880 1483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:35:37.168972 1483 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:35:37.169059 1483 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:35:37.179788 1483 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44817
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:35:37.181111 1483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:35:37.182485 1483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:35:37.195331 1489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:35:37.195299 1488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:35:37.195261 1491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:35:37.195660 1483 server_base.cc:1034] running on GCE node
I20260620 23:35:37.196722 1483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:35:37.198145 1483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:35:37.199388 1483 hybrid_clock.cc:648] HybridClock initialized: now 1781998537199332 us; error 37 us; skew 500 ppm
I20260620 23:35:37.199756 1483 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:35:37 dist-test-slave-tcgx krb5kdc[1470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:35:37.202770 1483 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 23:35:37.204281 1483 webserver.cc:460] Webserver started at http://127.0.97.62:38295/ using document root <none> and password file <none>
I20260620 23:35:37.205008 1483 fs_manager.cc:359] Metadata directory not provided
I20260620 23:35:37.205093 1483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:35:37.205351 1483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:35:37.208259 1483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/master-0/data/instance:
uuid: "b7bda5b0072b4a87b0297077f313def5"
format_stamp: "Formatted at 2026-06-20 23:35:37 on dist-test-slave-tcgx"
server_key: "8edb1316019c1c39a3c99273d5a00b8a"
server_key_iv: "490fdb4cf80ffc6a556a59d50a630092"
server_key_version: "encryptionkey@0"
I20260620 23:35:37.209005 1483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/master-0/wal/instance:
uuid: "b7bda5b0072b4a87b0297077f313def5"
format_stamp: "Formatted at 2026-06-20 23:35:37 on dist-test-slave-tcgx"
server_key: "8edb1316019c1c39a3c99273d5a00b8a"
server_key_iv: "490fdb4cf80ffc6a556a59d50a630092"
server_key_version: "encryptionkey@0"
I20260620 23:35:37.214061 1483 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260620 23:35:37.217703 1498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:35:37.219229 1483 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:35:37.219558 1483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/master-0/wal
uuid: "b7bda5b0072b4a87b0297077f313def5"
format_stamp: "Formatted at 2026-06-20 23:35:37 on dist-test-slave-tcgx"
server_key: "8edb1316019c1c39a3c99273d5a00b8a"
server_key_iv: "490fdb4cf80ffc6a556a59d50a630092"
server_key_version: "encryptionkey@0"
I20260620 23:35:37.219808 1483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:35:37.243031 1483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:35:37.247537 1483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:35:37.247814 1483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:35:37.269166 1483 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44817
I20260620 23:35:37.269186 1559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44817 every 8 connection(s)
I20260620 23:35:37.270567 1483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:35:37.274331 1560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:35:37.276685 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 1483
I20260620 23:35:37.276893 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/master-0/wal/instance
I20260620 23:35:37.277655 388 external_mini_cluster.cc:1428] Setting key a4f1393c2bb6361389e3b859ff8a21a0
I20260620 23:35:37.287294 1560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7bda5b0072b4a87b0297077f313def5: Bootstrap starting.
I20260620 23:35:37.291126 1560 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7bda5b0072b4a87b0297077f313def5: Neither blocks nor log segments found. Creating new log.
I20260620 23:35:37.292140 1560 log.cc:824] T 00000000000000000000000000000000 P b7bda5b0072b4a87b0297077f313def5: Log is configured to *not* fsync() on all Append() calls
Jun 20 23:35:37 dist-test-slave-tcgx krb5kdc[1470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:35:37.295192 1560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7bda5b0072b4a87b0297077f313def5: No bootstrap required, opened a new log
I20260620 23:35:37.303050 1563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:37.279996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60378 (local address 127.0.97.62:44817)
0620 23:35:37.280752 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:37.280768 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:37.280817 (+ 49us) server_negotiation.cc:407] Connection header received
0620 23:35:37.282990 (+ 2173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:37.283026 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:37.283583 (+ 557us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:37.284119 (+ 536us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:35:37.285415 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:37.287386 (+ 1971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:37.288721 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:37.289862 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:37.290045 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:37.294277 (+ 4232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:35:37.294288 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:35:37.294310 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:35:37.294354 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:35:37.298265 (+ 3911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:37.299011 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:37.299020 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:37.299029 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:37.299230 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:37.299660 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:37.299665 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:37.299668 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:37.300356 (+ 688us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:35:37.300656 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:37.301079 (+ 423us) server_negotiation.cc:299] Negotiation successful
0620 23:35:37.301556 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":307,"threads_started":1}
I20260620 23:35:37.305794 1560 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7bda5b0072b4a87b0297077f313def5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7bda5b0072b4a87b0297077f313def5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44817 } }
I20260620 23:35:37.306301 1560 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7bda5b0072b4a87b0297077f313def5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:35:37.306404 1560 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7bda5b0072b4a87b0297077f313def5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7bda5b0072b4a87b0297077f313def5, State: Initialized, Role: FOLLOWER
I20260620 23:35:37.306999 1560 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7bda5b0072b4a87b0297077f313def5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7bda5b0072b4a87b0297077f313def5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44817 } }
I20260620 23:35:37.307240 1560 raft_consensus.cc:388] T 00000000000000000000000000000000 P b7bda5b0072b4a87b0297077f313def5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:35:37.307389 1560 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7bda5b0072b4a87b0297077f313def5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:35:37.307570 1560 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7bda5b0072b4a87b0297077f313def5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:35:37.309696 1560 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7bda5b0072b4a87b0297077f313def5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7bda5b0072b4a87b0297077f313def5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44817 } }
I20260620 23:35:37.310228 1560 leader_election.cc:302] T 00000000000000000000000000000000 P b7bda5b0072b4a87b0297077f313def5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7bda5b0072b4a87b0297077f313def5; no voters:
I20260620 23:35:37.310809 1560 leader_election.cc:288] T 00000000000000000000000000000000 P b7bda5b0072b4a87b0297077f313def5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:35:37.311017 1565 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7bda5b0072b4a87b0297077f313def5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:35:37.312072 1565 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7bda5b0072b4a87b0297077f313def5 [term 1 LEADER]: Becoming Leader. State: Replica: b7bda5b0072b4a87b0297077f313def5, State: Running, Role: LEADER
I20260620 23:35:37.312606 1565 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7bda5b0072b4a87b0297077f313def5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7bda5b0072b4a87b0297077f313def5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44817 } }
I20260620 23:35:37.313020 1560 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7bda5b0072b4a87b0297077f313def5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:35:37.315742 1567 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7bda5b0072b4a87b0297077f313def5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7bda5b0072b4a87b0297077f313def5. Latest consensus state: current_term: 1 leader_uuid: "b7bda5b0072b4a87b0297077f313def5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7bda5b0072b4a87b0297077f313def5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44817 } } }
I20260620 23:35:37.316191 1567 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7bda5b0072b4a87b0297077f313def5 [sys.catalog]: This master's current role is: LEADER
I20260620 23:35:37.316887 1566 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7bda5b0072b4a87b0297077f313def5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b7bda5b0072b4a87b0297077f313def5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7bda5b0072b4a87b0297077f313def5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44817 } } }
I20260620 23:35:37.317232 1566 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7bda5b0072b4a87b0297077f313def5 [sys.catalog]: This master's current role is: LEADER
I20260620 23:35:37.322463 1570 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:35:37.328867 1570 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:35:37.338611 1570 catalog_manager.cc:1347] Generated new cluster ID: a8ca546ece424339a25c6c9fa02951c1
I20260620 23:35:37.338752 1570 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:35:37.367864 1570 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:35:37.369323 1570 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:35:37.375926 1570 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b7bda5b0072b4a87b0297077f313def5: Generated new TSK 0
I20260620 23:35:37.376968 1570 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 23:35:37.422463 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44817
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:35:37.754494 1586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:35:37.754803 1586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:35:37.754904 1586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:35:37.762238 1586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:35:37.762362 1586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:35:37.762449 1586 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 23:35:37.762530 1586 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 23:35:37.772629 1586 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:44817
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:35:37.773908 1586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:35:37.775293 1586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:35:37.789614 1591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:35:37.789577 1592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:35:37.789978 1594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:35:37.790638 1586 server_base.cc:1034] running on GCE node
I20260620 23:35:37.791425 1586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:35:37.792409 1586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:35:37.793643 1586 hybrid_clock.cc:648] HybridClock initialized: now 1781998537793570 us; error 47 us; skew 500 ppm
I20260620 23:35:37.793968 1586 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:35:37 dist-test-slave-tcgx krb5kdc[1470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:35:37.797231 1586 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 23:35:37.798758 1586 webserver.cc:460] Webserver started at http://127.0.97.1:34179/ using document root <none> and password file <none>
I20260620 23:35:37.799539 1586 fs_manager.cc:359] Metadata directory not provided
I20260620 23:35:37.799670 1586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:35:37.799999 1586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:35:37.803033 1586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/ts-0/data/instance:
uuid: "0ebfb31c16f54e308a4bdc944630bd61"
format_stamp: "Formatted at 2026-06-20 23:35:37 on dist-test-slave-tcgx"
server_key: "ef63e1f0e2b399c1a4f75fb156b2476c"
server_key_iv: "81789197a36ab5b0a1d092f871fcd6b7"
server_key_version: "encryptionkey@0"
I20260620 23:35:37.803855 1586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/ts-0/wal/instance:
uuid: "0ebfb31c16f54e308a4bdc944630bd61"
format_stamp: "Formatted at 2026-06-20 23:35:37 on dist-test-slave-tcgx"
server_key: "ef63e1f0e2b399c1a4f75fb156b2476c"
server_key_iv: "81789197a36ab5b0a1d092f871fcd6b7"
server_key_version: "encryptionkey@0"
I20260620 23:35:37.809340 1586 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:35:37.812949 1601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:35:37.814461 1586 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:35:37.814638 1586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/ts-0/wal
uuid: "0ebfb31c16f54e308a4bdc944630bd61"
format_stamp: "Formatted at 2026-06-20 23:35:37 on dist-test-slave-tcgx"
server_key: "ef63e1f0e2b399c1a4f75fb156b2476c"
server_key_iv: "81789197a36ab5b0a1d092f871fcd6b7"
server_key_version: "encryptionkey@0"
I20260620 23:35:37.814872 1586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:35:37.851899 1586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:35:37.856359 1586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:35:37.856652 1586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:35:37.859408 1586 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:35:37.859525 1586 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:35:37.859684 1586 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:35:37.859760 1586 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 23:35:37 dist-test-slave-tcgx krb5kdc[1470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:35:37.885418 1610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:37.860255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33157 (local address 127.0.97.62:44817)
0620 23:35:37.860805 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:37.860812 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:37.862112 (+ 1300us) server_negotiation.cc:407] Connection header received
0620 23:35:37.865489 (+ 3377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:37.865497 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:37.865620 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:37.865777 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:35:37.868231 (+ 2454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:37.869329 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:37.871882 (+ 2553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:37.872702 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:37.872870 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:37.877321 (+ 4451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:35:37.877331 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:35:37.877340 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:35:37.877396 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:35:37.880949 (+ 3553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:37.882286 (+ 1337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:37.882293 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:37.882297 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:37.882429 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:37.883062 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:37.883067 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:37.883070 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:37.883464 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:35:37.883621 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:37.884790 (+ 1169us) server_negotiation.cc:299] Negotiation successful
0620 23:35:37.884973 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":290,"threads_started":1}
I20260620 23:35:37.887365 1611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:37.860970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44817 (local address 127.0.97.1:33157)
0620 23:35:37.861892 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0620 23:35:37.861955 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 23:35:37.865019 (+ 3064us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:35:37.866107 (+ 1088us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:35:37.866124 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:35:37.867045 (+ 921us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:35:37.868016 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:37.868052 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:37.869485 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:37.869495 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:37.871634 (+ 2139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:37.871649 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:37.872944 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:37.872949 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:37.873145 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:37.873944 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:35:37.874001 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:35:37.877004 (+ 3003us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:35:37.881155 (+ 4151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:35:37.881169 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:35:37.881201 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:35:37.882107 (+ 906us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:35:37.882581 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:35:37.882586 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:35:37.882590 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:35:37.882929 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:35:37.883683 (+ 754us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:35:37.883690 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:35:37.884282 (+ 592us) client_negotiation.cc:769] Sending connection context
0620 23:35:37.885663 (+ 1381us) client_negotiation.cc:240] Negotiation successful
0620 23:35:37.886110 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":344,"threads_started":1}
I20260620 23:35:37.911810 1586 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42637
I20260620 23:35:37.911851 1721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42637 every 8 connection(s)
I20260620 23:35:37.913699 1586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
I20260620 23:35:37.916059 1722 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44817
I20260620 23:35:37.916872 1722 heartbeater.cc:464] Registering TS with master...
I20260620 23:35:37.917622 1722 heartbeater.cc:511] Master 127.0.97.62:44817 requested a full tablet report, sending...
I20260620 23:35:37.919936 1515 ts_manager.cc:194] Registered new tserver with Master: 0ebfb31c16f54e308a4bdc944630bd61 (127.0.97.1:42637)
I20260620 23:35:37.920608 1610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:37.903521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49877 (local address 127.0.97.62:44817)
0620 23:35:37.903743 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:37.903752 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:37.905703 (+ 1951us) server_negotiation.cc:407] Connection header received
0620 23:35:37.905969 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:37.905977 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:37.906073 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:37.906218 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:35:37.908020 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:37.909313 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:37.911585 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:37.912244 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:37.912348 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:37.914649 (+ 2301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:35:37.914655 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:35:37.914660 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:35:37.914711 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:35:37.917667 (+ 2956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:37.918436 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:37.918483 (+ 47us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:37.918486 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:37.918637 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:37.919066 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:37.919072 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:37.919076 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:37.919417 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:35:37.919593 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:37.919993 (+ 400us) server_negotiation.cc:299] Negotiation successful
0620 23:35:37.920219 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 23:35:37.920660 1697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:37.903523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44817 (local address 127.0.97.1:49877)
0620 23:35:37.905539 (+ 2016us) negotiation.cc:107] Waiting for socket to connect
0620 23:35:37.905572 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 23:35:37.905782 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:35:37.906275 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:35:37.906284 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:35:37.907238 (+ 954us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:35:37.907769 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:37.907792 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:37.909578 (+ 1786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:37.909587 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:37.911408 (+ 1821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:37.911425 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:37.912435 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:37.912441 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:37.912616 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:37.913286 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:35:37.913309 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:35:37.914450 (+ 1141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:35:37.917821 (+ 3371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:35:37.917827 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:35:37.917831 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:35:37.918300 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:35:37.918757 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:35:37.918764 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:35:37.918770 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:35:37.918899 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:35:37.919533 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:35:37.919538 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:35:37.919731 (+ 193us) client_negotiation.cc:769] Sending connection context
0620 23:35:37.919869 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 23:35:37.920212 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1816,"thread_start_us":1548,"threads_started":1}
I20260620 23:35:37.921722 1515 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:33157
I20260620 23:35:37.921994 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 1586
I20260620 23:35:37.922180 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/ts-0/wal/instance
I20260620 23:35:37.923113 388 external_mini_cluster.cc:1428] Setting key c549cbdac899b3eb8edd759b7c986d46
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 23:35:37.962016 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44817
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:35:38.291332 1726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:35:38.291642 1726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:35:38.291824 1726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:35:38.299495 1726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:35:38.299620 1726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:35:38.299759 1726 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 23:35:38.299888 1726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 23:35:38.310221 1726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:44817
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:35:38.311631 1726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:35:38.313036 1726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:35:38.327471 1731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:35:38.327502 1732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:35:38.327579 1734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:35:38.329021 1726 server_base.cc:1034] running on GCE node
I20260620 23:35:38.329681 1726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:35:38.330663 1726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:35:38.331878 1726 hybrid_clock.cc:648] HybridClock initialized: now 1781998538331780 us; error 74 us; skew 500 ppm
I20260620 23:35:38.332206 1726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:35:38 dist-test-slave-tcgx krb5kdc[1470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:35:38.335366 1726 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 23:35:38.336903 1726 webserver.cc:460] Webserver started at http://127.0.97.2:38635/ using document root <none> and password file <none>
I20260620 23:35:38.337654 1726 fs_manager.cc:359] Metadata directory not provided
I20260620 23:35:38.337787 1726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:35:38.338125 1726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:35:38.341339 1726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/ts-1/data/instance:
uuid: "8ba7348efe9d4a87bb7e0cfa35140f5a"
format_stamp: "Formatted at 2026-06-20 23:35:38 on dist-test-slave-tcgx"
server_key: "50d995339095914cb8af755cbc03796e"
server_key_iv: "5594527335477521e90e4183c5229090"
server_key_version: "encryptionkey@0"
I20260620 23:35:38.342190 1726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/ts-1/wal/instance:
uuid: "8ba7348efe9d4a87bb7e0cfa35140f5a"
format_stamp: "Formatted at 2026-06-20 23:35:38 on dist-test-slave-tcgx"
server_key: "50d995339095914cb8af755cbc03796e"
server_key_iv: "5594527335477521e90e4183c5229090"
server_key_version: "encryptionkey@0"
I20260620 23:35:38.347604 1726 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 23:35:38.351475 1741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:35:38.353032 1726 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 23:35:38.353237 1726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/ts-1/wal
uuid: "8ba7348efe9d4a87bb7e0cfa35140f5a"
format_stamp: "Formatted at 2026-06-20 23:35:38 on dist-test-slave-tcgx"
server_key: "50d995339095914cb8af755cbc03796e"
server_key_iv: "5594527335477521e90e4183c5229090"
server_key_version: "encryptionkey@0"
I20260620 23:35:38.353480 1726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:35:38.366771 1726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:35:38.371171 1726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:35:38.371502 1726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:35:38.374150 1726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:35:38.374276 1726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:35:38.374442 1726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:35:38.374528 1726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 23:35:38 dist-test-slave-tcgx krb5kdc[1470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:35:38.400934 1610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:38.375177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58171 (local address 127.0.97.62:44817)
0620 23:35:38.375406 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:38.375415 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:38.377099 (+ 1684us) server_negotiation.cc:407] Connection header received
0620 23:35:38.380576 (+ 3477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:38.380593 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:38.380688 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:38.380824 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:35:38.383552 (+ 2728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:38.384313 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:38.387610 (+ 3297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:38.388241 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:38.388392 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:38.392970 (+ 4578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:35:38.392975 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:35:38.392979 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:35:38.393043 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:35:38.396195 (+ 3152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:38.397364 (+ 1169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:38.397369 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:38.397372 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:38.397537 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:38.398296 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:38.398304 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:38.398310 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:38.398787 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:35:38.398995 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:38.400326 (+ 1331us) server_negotiation.cc:299] Negotiation successful
0620 23:35:38.400564 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 23:35:38.402077 1750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:38.375893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44817 (local address 127.0.97.2:58171)
0620 23:35:38.376872 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0620 23:35:38.376923 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 23:35:38.380152 (+ 3229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:35:38.381147 (+ 995us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:35:38.381166 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:35:38.382112 (+ 946us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:35:38.383236 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:38.383312 (+ 76us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:38.384489 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:38.384498 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:38.387398 (+ 2900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:38.387421 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:38.388543 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:38.388552 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:38.388790 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:38.389532 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:35:38.389556 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:35:38.392668 (+ 3112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:35:38.396358 (+ 3690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:35:38.396370 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:35:38.396396 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:35:38.397159 (+ 763us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:35:38.397693 (+ 534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:35:38.397702 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:35:38.397709 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:35:38.398097 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:35:38.399128 (+ 1031us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:35:38.399140 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:35:38.399798 (+ 658us) client_negotiation.cc:769] Sending connection context
0620 23:35:38.400230 (+ 432us) client_negotiation.cc:240] Negotiation successful
0620 23:35:38.400826 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":442,"threads_started":1}
I20260620 23:35:38.425019 1610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:38.408994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55721 (local address 127.0.97.62:44817)
0620 23:35:38.409239 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:38.409248 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:38.409596 (+ 348us) server_negotiation.cc:407] Connection header received
0620 23:35:38.409832 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:38.409841 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:38.409947 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:38.410104 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:35:38.411118 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:38.412612 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:38.414793 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:38.415310 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:38.415469 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:38.418008 (+ 2539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:35:38.418013 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:35:38.418016 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:35:38.418050 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:35:38.421526 (+ 3476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:38.422442 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:38.422452 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:38.422457 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:38.422641 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:38.423125 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:38.423133 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:38.423139 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:38.423697 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:35:38.423922 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:38.424388 (+ 466us) server_negotiation.cc:299] Negotiation successful
0620 23:35:38.424644 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 23:35:38.425045 1810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:38.408645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44817 (local address 127.0.97.2:55721)
0620 23:35:38.409466 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0620 23:35:38.409487 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:35:38.409685 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:35:38.410124 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:35:38.410127 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:35:38.410573 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:35:38.410884 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:38.410893 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:38.412815 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:38.412822 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:38.414607 (+ 1785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:38.414621 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:38.415521 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:38.415528 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:38.415746 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:38.416567 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:35:38.416599 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:35:38.417782 (+ 1183us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:35:38.421684 (+ 3902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:35:38.421692 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:35:38.421698 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:35:38.422275 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:35:38.422777 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:35:38.422782 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:35:38.422786 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:35:38.422997 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:35:38.423854 (+ 857us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:35:38.423862 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:35:38.424140 (+ 278us) client_negotiation.cc:769] Sending connection context
0620 23:35:38.424327 (+ 187us) client_negotiation.cc:240] Negotiation successful
0620 23:35:38.424650 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"thread_start_us":524,"threads_started":1}
I20260620 23:35:38.435180 1726 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34037
I20260620 23:35:38.435211 1860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34037 every 8 connection(s)
I20260620 23:35:38.436579 1726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
I20260620 23:35:38.439185 1861 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44817
I20260620 23:35:38.439438 1861 heartbeater.cc:464] Registering TS with master...
I20260620 23:35:38.440127 1861 heartbeater.cc:511] Master 127.0.97.62:44817 requested a full tablet report, sending...
I20260620 23:35:38.440621 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 1726
I20260620 23:35:38.440794 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/ts-1/wal/instance
I20260620 23:35:38.441293 388 external_mini_cluster.cc:1428] Setting key 7af3bf19babfbb6692855f7696295344
I20260620 23:35:38.441776 1514 ts_manager.cc:194] Registered new tserver with Master: 8ba7348efe9d4a87bb7e0cfa35140f5a (127.0.97.2:34037)
I20260620 23:35:38.442759 1514 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:58171
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 23:35:38.482393 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44817
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:35:38.811329 1865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:35:38.811635 1865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:35:38.811810 1865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:35:38.819234 1865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:35:38.819355 1865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:35:38.819525 1865 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 23:35:38.819687 1865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 23:35:38.829900 1865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:44817
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:35:38.831408 1865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:35:38.832760 1865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:35:38.846411 1871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:35:38.846462 1870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:35:38.847108 1873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:35:38.847206 1865 server_base.cc:1034] running on GCE node
I20260620 23:35:38.847916 1865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:35:38.848923 1865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:35:38.850155 1865 hybrid_clock.cc:648] HybridClock initialized: now 1781998538850089 us; error 38 us; skew 500 ppm
I20260620 23:35:38.850512 1865 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:35:38 dist-test-slave-tcgx krb5kdc[1470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:35:38.858568 1865 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 23:35:38.860271 1865 webserver.cc:460] Webserver started at http://127.0.97.3:41485/ using document root <none> and password file <none>
I20260620 23:35:38.861053 1865 fs_manager.cc:359] Metadata directory not provided
I20260620 23:35:38.861168 1865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:35:38.861526 1865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:35:38.864717 1865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/ts-2/data/instance:
uuid: "e85eb50b63bd4eb09824856f7241940a"
format_stamp: "Formatted at 2026-06-20 23:35:38 on dist-test-slave-tcgx"
server_key: "2cc0d33c0dfdf7607854c2c64d46261f"
server_key_iv: "b5d95f72091b2d6190ee68f48c6ea9dc"
server_key_version: "encryptionkey@0"
I20260620 23:35:38.865520 1865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/ts-2/wal/instance:
uuid: "e85eb50b63bd4eb09824856f7241940a"
format_stamp: "Formatted at 2026-06-20 23:35:38 on dist-test-slave-tcgx"
server_key: "2cc0d33c0dfdf7607854c2c64d46261f"
server_key_iv: "b5d95f72091b2d6190ee68f48c6ea9dc"
server_key_version: "encryptionkey@0"
I20260620 23:35:38.870903 1865 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 23:35:38.874557 1880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:35:38.876055 1865 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:35:38.876235 1865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/ts-2/wal
uuid: "e85eb50b63bd4eb09824856f7241940a"
format_stamp: "Formatted at 2026-06-20 23:35:38 on dist-test-slave-tcgx"
server_key: "2cc0d33c0dfdf7607854c2c64d46261f"
server_key_iv: "b5d95f72091b2d6190ee68f48c6ea9dc"
server_key_version: "encryptionkey@0"
I20260620 23:35:38.876477 1865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:35:38.896576 1865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:35:38.901086 1865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:35:38.901376 1865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:35:38.904304 1865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:35:38.904438 1865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:35:38.904597 1865 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:35:38.904666 1865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 23:35:38 dist-test-slave-tcgx krb5kdc[1470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:35:38.926043 1722 heartbeater.cc:503] Master 127.0.97.62:44817 was elected leader, sending a full tablet report...
I20260620 23:35:38.930805 1610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:38.904960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42801 (local address 127.0.97.62:44817)
0620 23:35:38.905177 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:38.905183 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:38.906730 (+ 1547us) server_negotiation.cc:407] Connection header received
0620 23:35:38.909798 (+ 3068us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:38.909807 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:38.909896 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:38.910005 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:35:38.912852 (+ 2847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:38.914285 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:38.917594 (+ 3309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:38.918157 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:38.918328 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:38.924048 (+ 5720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:35:38.924053 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:35:38.924056 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:35:38.924092 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:35:38.927539 (+ 3447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:38.928446 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:38.928449 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:38.928451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:38.928573 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:38.929023 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:38.929028 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:38.929031 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:38.929316 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:35:38.929469 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:38.930213 (+ 744us) server_negotiation.cc:299] Negotiation successful
0620 23:35:38.930433 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 23:35:38.932440 1889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:38.905598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44817 (local address 127.0.97.3:42801)
0620 23:35:38.906512 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0620 23:35:38.906582 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0620 23:35:38.909375 (+ 2793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:35:38.910357 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:35:38.910370 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:35:38.911387 (+ 1017us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:35:38.912538 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:38.912572 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:38.914517 (+ 1945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:38.914529 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:38.917392 (+ 2863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:38.917413 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:38.918596 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:38.918601 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:38.918899 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:38.919770 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:35:38.919816 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:35:38.923741 (+ 3925us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:35:38.927726 (+ 3985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:35:38.927738 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:35:38.927759 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:35:38.928303 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:35:38.928699 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:35:38.928703 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:35:38.928707 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:35:38.928900 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:35:38.929445 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:35:38.929453 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:35:38.929840 (+ 387us) client_negotiation.cc:769] Sending connection context
0620 23:35:38.930118 (+ 278us) client_negotiation.cc:240] Negotiation successful
0620 23:35:38.930709 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":399,"threads_started":1}
I20260620 23:35:38.955415 1610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:38.939091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60299 (local address 127.0.97.62:44817)
0620 23:35:38.939315 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:38.939323 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:38.940066 (+ 743us) server_negotiation.cc:407] Connection header received
0620 23:35:38.940239 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:38.940247 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:38.940352 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:38.940492 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:35:38.942158 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:38.942867 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:38.945455 (+ 2588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:38.945951 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:38.946103 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:38.948840 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:35:38.948848 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:35:38.948855 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:35:38.948899 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:35:38.952164 (+ 3265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:38.953036 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:38.953044 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:38.953048 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:38.953188 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:38.953560 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:38.953567 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:38.953570 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:38.953843 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:35:38.954011 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:38.954897 (+ 886us) server_negotiation.cc:299] Negotiation successful
0620 23:35:38.955183 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 23:35:38.955569 1948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:38.939109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44817 (local address 127.0.97.3:60299)
0620 23:35:38.939952 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0620 23:35:38.939968 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:35:38.940100 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:35:38.940718 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:35:38.940726 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:35:38.941475 (+ 749us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:35:38.941964 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:38.941981 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:38.943076 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:38.943084 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:38.945267 (+ 2183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:38.945292 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:38.946192 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:38.946202 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:38.946469 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:38.947408 (+ 939us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:35:38.947439 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:35:38.948570 (+ 1131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:35:38.952309 (+ 3739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:35:38.952316 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:35:38.952322 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:35:38.952907 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:35:38.953303 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:35:38.953307 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:35:38.953311 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:35:38.953445 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:35:38.953963 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:35:38.953968 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:35:38.954215 (+ 247us) client_negotiation.cc:769] Sending connection context
0620 23:35:38.954433 (+ 218us) client_negotiation.cc:240] Negotiation successful
0620 23:35:38.954852 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"thread_start_us":462,"threads_started":1}
I20260620 23:35:38.968837 1865 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43707
I20260620 23:35:38.968866 1999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43707 every 8 connection(s)
I20260620 23:35:38.970175 1865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
I20260620 23:35:38.971424 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 1865
I20260620 23:35:38.971606 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/ts-2/wal/instance
I20260620 23:35:38.972119 388 external_mini_cluster.cc:1428] Setting key 06eaf91627d7dd4a527ee8ec676c0c35
I20260620 23:35:38.972501 2000 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44817
I20260620 23:35:38.972747 2000 heartbeater.cc:464] Registering TS with master...
I20260620 23:35:38.973640 2000 heartbeater.cc:511] Master 127.0.97.62:44817 requested a full tablet report, sending...
I20260620 23:35:38.975215 1515 ts_manager.cc:194] Registered new tserver with Master: e85eb50b63bd4eb09824856f7241940a (127.0.97.3:43707)
I20260620 23:35:38.976034 1515 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:42801
I20260620 23:35:38.987848 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 23:35:39 dist-test-slave-tcgx krb5kdc[1470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998539, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 23:35:39 dist-test-slave-tcgx krb5kdc[1470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998539, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:35:39.034701 1610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:39.020167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60390 (local address 127.0.97.62:44817)
0620 23:35:39.020382 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:39.020390 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:39.020682 (+ 292us) server_negotiation.cc:407] Connection header received
0620 23:35:39.020904 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:39.020909 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:39.020992 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:39.021098 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:35:39.022475 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:39.023415 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:39.024896 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:39.025351 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:39.025490 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:39.029468 (+ 3978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:35:39.029473 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:35:39.029476 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:35:39.029513 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:35:39.032023 (+ 2510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:39.032710 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:39.032718 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:39.032721 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:39.032853 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:39.033238 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:39.033245 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:39.033248 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:39.033520 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:35:39.033769 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:39.034084 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 23:35:39.034359 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 23:35:39.038834 1515 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60390:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:35:39.046211 1515 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:35:39.081092 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:39.058898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42637 (local address 127.0.0.1:57654)
0620 23:35:39.071309 (+ 12411us) negotiation.cc:107] Waiting for socket to connect
0620 23:35:39.071368 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 23:35:39.071570 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:35:39.073045 (+ 1475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:35:39.073056 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:35:39.073100 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:35:39.073631 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:39.073659 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:39.075606 (+ 1947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:39.075612 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:39.077883 (+ 2271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:39.077899 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:39.079894 (+ 1995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:39.079901 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:39.080168 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:39.080245 (+ 77us) client_negotiation.cc:769] Sending connection context
0620 23:35:39.080385 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 23:35:39.080584 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12132,"mutex_wait_us":3,"thread_start_us":15135,"threads_started":1}
I20260620 23:35:39.081948 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:39.058974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57654 (local address 127.0.97.1:42637)
0620 23:35:39.072050 (+ 13076us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:39.072061 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:39.072095 (+ 34us) server_negotiation.cc:407] Connection header received
0620 23:35:39.072219 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:39.072227 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:39.072500 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:39.072715 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:35:39.073879 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:39.075466 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:39.078059 (+ 2593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:39.079688 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:39.079865 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:39.080159 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:39.081445 (+ 1286us) server_negotiation.cc:299] Negotiation successful
0620 23:35:39.081644 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12886,"thread_start_us":12768,"threads_started":1}
I20260620 23:35:39.082516 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:39.061238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43707 (local address 127.0.0.1:46230)
0620 23:35:39.074344 (+ 13106us) negotiation.cc:107] Waiting for socket to connect
0620 23:35:39.074373 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 23:35:39.074482 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:35:39.075108 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:35:39.075112 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:35:39.075136 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:35:39.075562 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:39.075575 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:39.077622 (+ 2047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:39.077627 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:39.079447 (+ 1820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:39.079461 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:39.081642 (+ 2181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:39.081648 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:39.081815 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:39.081851 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 23:35:39.081935 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 23:35:39.082092 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12967,"mutex_wait_us":75,"thread_start_us":10882,"threads_started":1}
I20260620 23:35:39.082899 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:39.059531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53742 (local address 127.0.97.2:34037)
0620 23:35:39.074625 (+ 15094us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:39.074633 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:39.074662 (+ 29us) server_negotiation.cc:407] Connection header received
0620 23:35:39.074750 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:39.074757 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:39.075016 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:39.075210 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:35:39.076275 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:39.077610 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:39.080134 (+ 2524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:39.081510 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:39.081648 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:39.081865 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:39.082450 (+ 585us) server_negotiation.cc:299] Negotiation successful
0620 23:35:39.082622 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14947,"thread_start_us":14832,"threads_started":1}
I20260620 23:35:39.083555 2018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:39.061548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46230 (local address 127.0.97.3:43707)
0620 23:35:39.069766 (+ 8218us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:39.069780 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:39.074603 (+ 4823us) server_negotiation.cc:407] Connection header received
0620 23:35:39.074701 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:39.074706 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:39.074883 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:39.075124 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:35:39.075773 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:39.077483 (+ 1710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:39.079616 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:39.081141 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:39.082797 (+ 1656us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:39.083087 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:39.083159 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 23:35:39.083317 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7962,"thread_start_us":7714,"threads_started":1}
I20260620 23:35:39.082921 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:39.059036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34037 (local address 127.0.0.1:53742)
0620 23:35:39.072365 (+ 13329us) negotiation.cc:107] Waiting for socket to connect
0620 23:35:39.072388 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 23:35:39.072509 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:35:39.075400 (+ 2891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:35:39.075405 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:35:39.075435 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:35:39.076041 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:39.076057 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:39.077750 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:39.077756 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:39.079955 (+ 2199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:39.079970 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:39.082017 (+ 2047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:39.082024 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:39.082233 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:39.082267 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 23:35:39.082401 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 23:35:39.082541 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13218,"mutex_wait_us":30,"thread_start_us":11847,"threads_started":1}
I20260620 23:35:39.086223 1652 tablet_service.cc:1467] Processing CreateTablet for tablet 66f17c8828054dacb83c5f1343363424 (DEFAULT_TABLE table=test-table [id=e89b125b2b774d01b22a7c295ff522b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:35:39.086690 1791 tablet_service.cc:1467] Processing CreateTablet for tablet 66f17c8828054dacb83c5f1343363424 (DEFAULT_TABLE table=test-table [id=e89b125b2b774d01b22a7c295ff522b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:35:39.087148 1930 tablet_service.cc:1467] Processing CreateTablet for tablet 66f17c8828054dacb83c5f1343363424 (DEFAULT_TABLE table=test-table [id=e89b125b2b774d01b22a7c295ff522b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:35:39.087824 1652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66f17c8828054dacb83c5f1343363424. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:35:39.087942 1791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66f17c8828054dacb83c5f1343363424. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:35:39.088439 1930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66f17c8828054dacb83c5f1343363424. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:35:39.099867 2019 tablet_bootstrap.cc:492] T 66f17c8828054dacb83c5f1343363424 P 8ba7348efe9d4a87bb7e0cfa35140f5a: Bootstrap starting.
I20260620 23:35:39.103215 2021 tablet_bootstrap.cc:492] T 66f17c8828054dacb83c5f1343363424 P e85eb50b63bd4eb09824856f7241940a: Bootstrap starting.
I20260620 23:35:39.103832 2019 tablet_bootstrap.cc:654] T 66f17c8828054dacb83c5f1343363424 P 8ba7348efe9d4a87bb7e0cfa35140f5a: Neither blocks nor log segments found. Creating new log.
I20260620 23:35:39.105723 2019 log.cc:824] T 66f17c8828054dacb83c5f1343363424 P 8ba7348efe9d4a87bb7e0cfa35140f5a: Log is configured to *not* fsync() on all Append() calls
I20260620 23:35:39.107125 2021 tablet_bootstrap.cc:654] T 66f17c8828054dacb83c5f1343363424 P e85eb50b63bd4eb09824856f7241940a: Neither blocks nor log segments found. Creating new log.
I20260620 23:35:39.108399 2021 log.cc:824] T 66f17c8828054dacb83c5f1343363424 P e85eb50b63bd4eb09824856f7241940a: Log is configured to *not* fsync() on all Append() calls
I20260620 23:35:39.109236 2020 tablet_bootstrap.cc:492] T 66f17c8828054dacb83c5f1343363424 P 0ebfb31c16f54e308a4bdc944630bd61: Bootstrap starting.
I20260620 23:35:39.109344 2019 tablet_bootstrap.cc:492] T 66f17c8828054dacb83c5f1343363424 P 8ba7348efe9d4a87bb7e0cfa35140f5a: No bootstrap required, opened a new log
I20260620 23:35:39.109633 2019 ts_tablet_manager.cc:1397] T 66f17c8828054dacb83c5f1343363424 P 8ba7348efe9d4a87bb7e0cfa35140f5a: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.008s
I20260620 23:35:39.111063 2021 tablet_bootstrap.cc:492] T 66f17c8828054dacb83c5f1343363424 P e85eb50b63bd4eb09824856f7241940a: No bootstrap required, opened a new log
I20260620 23:35:39.111337 2021 ts_tablet_manager.cc:1397] T 66f17c8828054dacb83c5f1343363424 P e85eb50b63bd4eb09824856f7241940a: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.003s
I20260620 23:35:39.112884 2020 tablet_bootstrap.cc:654] T 66f17c8828054dacb83c5f1343363424 P 0ebfb31c16f54e308a4bdc944630bd61: Neither blocks nor log segments found. Creating new log.
I20260620 23:35:39.114197 2020 log.cc:824] T 66f17c8828054dacb83c5f1343363424 P 0ebfb31c16f54e308a4bdc944630bd61: Log is configured to *not* fsync() on all Append() calls
I20260620 23:35:39.119848 2021 raft_consensus.cc:348] T 66f17c8828054dacb83c5f1343363424 P e85eb50b63bd4eb09824856f7241940a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebfb31c16f54e308a4bdc944630bd61" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42637 } } peers { permanent_uuid: "8ba7348efe9d4a87bb7e0cfa35140f5a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34037 } } peers { permanent_uuid: "e85eb50b63bd4eb09824856f7241940a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43707 } }
I20260620 23:35:39.120370 2021 raft_consensus.cc:374] T 66f17c8828054dacb83c5f1343363424 P e85eb50b63bd4eb09824856f7241940a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:35:39.120471 2021 raft_consensus.cc:729] T 66f17c8828054dacb83c5f1343363424 P e85eb50b63bd4eb09824856f7241940a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e85eb50b63bd4eb09824856f7241940a, State: Initialized, Role: FOLLOWER
I20260620 23:35:39.121143 2021 consensus_queue.cc:260] T 66f17c8828054dacb83c5f1343363424 P e85eb50b63bd4eb09824856f7241940a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebfb31c16f54e308a4bdc944630bd61" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42637 } } peers { permanent_uuid: "8ba7348efe9d4a87bb7e0cfa35140f5a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34037 } } peers { permanent_uuid: "e85eb50b63bd4eb09824856f7241940a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43707 } }
I20260620 23:35:39.122331 2000 heartbeater.cc:503] Master 127.0.97.62:44817 was elected leader, sending a full tablet report...
I20260620 23:35:39.122732 2019 raft_consensus.cc:348] T 66f17c8828054dacb83c5f1343363424 P 8ba7348efe9d4a87bb7e0cfa35140f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebfb31c16f54e308a4bdc944630bd61" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42637 } } peers { permanent_uuid: "8ba7348efe9d4a87bb7e0cfa35140f5a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34037 } } peers { permanent_uuid: "e85eb50b63bd4eb09824856f7241940a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43707 } }
I20260620 23:35:39.123229 2019 raft_consensus.cc:374] T 66f17c8828054dacb83c5f1343363424 P 8ba7348efe9d4a87bb7e0cfa35140f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:35:39.123339 2019 raft_consensus.cc:729] T 66f17c8828054dacb83c5f1343363424 P 8ba7348efe9d4a87bb7e0cfa35140f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ba7348efe9d4a87bb7e0cfa35140f5a, State: Initialized, Role: FOLLOWER
I20260620 23:35:39.123570 2021 ts_tablet_manager.cc:1428] T 66f17c8828054dacb83c5f1343363424 P e85eb50b63bd4eb09824856f7241940a: Time spent starting tablet: real 0.012s user 0.010s sys 0.001s
I20260620 23:35:39.124182 2019 consensus_queue.cc:260] T 66f17c8828054dacb83c5f1343363424 P 8ba7348efe9d4a87bb7e0cfa35140f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebfb31c16f54e308a4bdc944630bd61" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42637 } } peers { permanent_uuid: "8ba7348efe9d4a87bb7e0cfa35140f5a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34037 } } peers { permanent_uuid: "e85eb50b63bd4eb09824856f7241940a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43707 } }
I20260620 23:35:39.124965 2020 tablet_bootstrap.cc:492] T 66f17c8828054dacb83c5f1343363424 P 0ebfb31c16f54e308a4bdc944630bd61: No bootstrap required, opened a new log
I20260620 23:35:39.125329 2020 ts_tablet_manager.cc:1397] T 66f17c8828054dacb83c5f1343363424 P 0ebfb31c16f54e308a4bdc944630bd61: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.009s
I20260620 23:35:39.128124 1861 heartbeater.cc:503] Master 127.0.97.62:44817 was elected leader, sending a full tablet report...
I20260620 23:35:39.128813 2019 ts_tablet_manager.cc:1428] T 66f17c8828054dacb83c5f1343363424 P 8ba7348efe9d4a87bb7e0cfa35140f5a: Time spent starting tablet: real 0.019s user 0.015s sys 0.003s
I20260620 23:35:39.134080 2020 raft_consensus.cc:348] T 66f17c8828054dacb83c5f1343363424 P 0ebfb31c16f54e308a4bdc944630bd61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebfb31c16f54e308a4bdc944630bd61" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42637 } } peers { permanent_uuid: "8ba7348efe9d4a87bb7e0cfa35140f5a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34037 } } peers { permanent_uuid: "e85eb50b63bd4eb09824856f7241940a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43707 } }
I20260620 23:35:39.134485 2020 raft_consensus.cc:374] T 66f17c8828054dacb83c5f1343363424 P 0ebfb31c16f54e308a4bdc944630bd61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:35:39.134604 2020 raft_consensus.cc:729] T 66f17c8828054dacb83c5f1343363424 P 0ebfb31c16f54e308a4bdc944630bd61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ebfb31c16f54e308a4bdc944630bd61, State: Initialized, Role: FOLLOWER
I20260620 23:35:39.135293 2020 consensus_queue.cc:260] T 66f17c8828054dacb83c5f1343363424 P 0ebfb31c16f54e308a4bdc944630bd61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebfb31c16f54e308a4bdc944630bd61" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42637 } } peers { permanent_uuid: "8ba7348efe9d4a87bb7e0cfa35140f5a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34037 } } peers { permanent_uuid: "e85eb50b63bd4eb09824856f7241940a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43707 } }
I20260620 23:35:39.136966 2020 ts_tablet_manager.cc:1428] T 66f17c8828054dacb83c5f1343363424 P 0ebfb31c16f54e308a4bdc944630bd61: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
W20260620 23:35:39.168098 1723 tablet.cc:2369] T 66f17c8828054dacb83c5f1343363424 P 0ebfb31c16f54e308a4bdc944630bd61: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:35:39.188763 1862 tablet.cc:2369] T 66f17c8828054dacb83c5f1343363424 P 8ba7348efe9d4a87bb7e0cfa35140f5a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:35:39.221933 2001 tablet.cc:2369] T 66f17c8828054dacb83c5f1343363424 P e85eb50b63bd4eb09824856f7241940a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:35:39.508535 2025 raft_consensus.cc:482] T 66f17c8828054dacb83c5f1343363424 P e85eb50b63bd4eb09824856f7241940a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:35:39.508844 2025 raft_consensus.cc:504] T 66f17c8828054dacb83c5f1343363424 P e85eb50b63bd4eb09824856f7241940a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebfb31c16f54e308a4bdc944630bd61" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42637 } } peers { permanent_uuid: "8ba7348efe9d4a87bb7e0cfa35140f5a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34037 } } peers { permanent_uuid: "e85eb50b63bd4eb09824856f7241940a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43707 } }
I20260620 23:35:39.511171 2025 leader_election.cc:288] T 66f17c8828054dacb83c5f1343363424 P e85eb50b63bd4eb09824856f7241940a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ebfb31c16f54e308a4bdc944630bd61 (127.0.97.1:42637), 8ba7348efe9d4a87bb7e0cfa35140f5a (127.0.97.2:34037)
I20260620 23:35:39.519755 2029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:39.511482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42637 (local address 127.0.97.3:53521)
0620 23:35:39.512099 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0620 23:35:39.512120 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:35:39.512290 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:35:39.513596 (+ 1306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:35:39.513601 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:35:39.513626 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:35:39.513883 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:39.513892 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:39.515400 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:39.515406 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:39.517472 (+ 2066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:39.517485 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:39.518809 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:39.518817 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:39.519050 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:39.519087 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 23:35:39.519192 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 23:35:39.519375 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"mutex_wait_us":55,"thread_start_us":1728,"threads_started":1}
I20260620 23:35:39.520089 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:39.513161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55963 (local address 127.0.97.2:34037)
0620 23:35:39.513360 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:39.513367 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:39.513386 (+ 19us) server_negotiation.cc:407] Connection header received
0620 23:35:39.513456 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:39.513461 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:39.513531 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:39.513625 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:35:39.514459 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:39.515626 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:39.517982 (+ 2356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:39.518901 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:39.519084 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:39.519190 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:39.519661 (+ 471us) server_negotiation.cc:299] Negotiation successful
0620 23:35:39.519822 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 23:35:39.520219 2028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:39.511503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34037 (local address 127.0.97.3:55963)
0620 23:35:39.512780 (+ 1277us) negotiation.cc:107] Waiting for socket to connect
0620 23:35:39.512798 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:35:39.512899 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:35:39.513786 (+ 887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:35:39.513795 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:35:39.513819 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:35:39.514256 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:39.514271 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:39.515782 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:39.515787 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:39.517745 (+ 1958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:39.517761 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:39.519150 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:39.519222 (+ 72us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:39.519461 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:39.519501 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 23:35:39.519630 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 23:35:39.519775 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1152,"mutex_wait_us":4,"thread_start_us":993,"threads_started":1}
I20260620 23:35:39.520203 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:39.513044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53521 (local address 127.0.97.1:42637)
0620 23:35:39.513266 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:39.513272 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:39.513291 (+ 19us) server_negotiation.cc:407] Connection header received
0620 23:35:39.513366 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:39.513371 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:39.513458 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:39.513569 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:35:39.514106 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:39.515252 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:39.517645 (+ 2393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:39.518630 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:39.518742 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:39.518853 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:39.519287 (+ 434us) server_negotiation.cc:299] Negotiation successful
0620 23:35:39.519459 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260620 23:35:39.521340 1816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66f17c8828054dacb83c5f1343363424" candidate_uuid: "e85eb50b63bd4eb09824856f7241940a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ba7348efe9d4a87bb7e0cfa35140f5a" is_pre_election: true
I20260620 23:35:39.521340 1672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66f17c8828054dacb83c5f1343363424" candidate_uuid: "e85eb50b63bd4eb09824856f7241940a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ebfb31c16f54e308a4bdc944630bd61" is_pre_election: true
I20260620 23:35:39.521939 1672 raft_consensus.cc:2454] T 66f17c8828054dacb83c5f1343363424 P 0ebfb31c16f54e308a4bdc944630bd61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e85eb50b63bd4eb09824856f7241940a in term 0.
I20260620 23:35:39.521939 1816 raft_consensus.cc:2454] T 66f17c8828054dacb83c5f1343363424 P 8ba7348efe9d4a87bb7e0cfa35140f5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e85eb50b63bd4eb09824856f7241940a in term 0.
I20260620 23:35:39.522671 1883 leader_election.cc:302] T 66f17c8828054dacb83c5f1343363424 P e85eb50b63bd4eb09824856f7241940a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ba7348efe9d4a87bb7e0cfa35140f5a, e85eb50b63bd4eb09824856f7241940a; no voters:
I20260620 23:35:39.523129 2025 raft_consensus.cc:2781] T 66f17c8828054dacb83c5f1343363424 P e85eb50b63bd4eb09824856f7241940a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:35:39.523299 2025 raft_consensus.cc:482] T 66f17c8828054dacb83c5f1343363424 P e85eb50b63bd4eb09824856f7241940a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:35:39.523418 2025 raft_consensus.cc:3037] T 66f17c8828054dacb83c5f1343363424 P e85eb50b63bd4eb09824856f7241940a [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:35:39.530120 2025 raft_consensus.cc:504] T 66f17c8828054dacb83c5f1343363424 P e85eb50b63bd4eb09824856f7241940a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebfb31c16f54e308a4bdc944630bd61" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42637 } } peers { permanent_uuid: "8ba7348efe9d4a87bb7e0cfa35140f5a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34037 } } peers { permanent_uuid: "e85eb50b63bd4eb09824856f7241940a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43707 } }
I20260620 23:35:39.530861 2025 leader_election.cc:288] T 66f17c8828054dacb83c5f1343363424 P e85eb50b63bd4eb09824856f7241940a [CANDIDATE]: Term 1 election: Requested vote from peers 0ebfb31c16f54e308a4bdc944630bd61 (127.0.97.1:42637), 8ba7348efe9d4a87bb7e0cfa35140f5a (127.0.97.2:34037)
I20260620 23:35:39.531492 1672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66f17c8828054dacb83c5f1343363424" candidate_uuid: "e85eb50b63bd4eb09824856f7241940a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ebfb31c16f54e308a4bdc944630bd61"
I20260620 23:35:39.531544 1816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66f17c8828054dacb83c5f1343363424" candidate_uuid: "e85eb50b63bd4eb09824856f7241940a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ba7348efe9d4a87bb7e0cfa35140f5a"
I20260620 23:35:39.531751 1672 raft_consensus.cc:3037] T 66f17c8828054dacb83c5f1343363424 P 0ebfb31c16f54e308a4bdc944630bd61 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:35:39.531811 1816 raft_consensus.cc:3037] T 66f17c8828054dacb83c5f1343363424 P 8ba7348efe9d4a87bb7e0cfa35140f5a [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:35:39.535076 1816 raft_consensus.cc:2454] T 66f17c8828054dacb83c5f1343363424 P 8ba7348efe9d4a87bb7e0cfa35140f5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e85eb50b63bd4eb09824856f7241940a in term 1.
I20260620 23:35:39.535167 1672 raft_consensus.cc:2454] T 66f17c8828054dacb83c5f1343363424 P 0ebfb31c16f54e308a4bdc944630bd61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e85eb50b63bd4eb09824856f7241940a in term 1.
I20260620 23:35:39.535568 1883 leader_election.cc:302] T 66f17c8828054dacb83c5f1343363424 P e85eb50b63bd4eb09824856f7241940a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ba7348efe9d4a87bb7e0cfa35140f5a, e85eb50b63bd4eb09824856f7241940a; no voters:
I20260620 23:35:39.535916 2025 raft_consensus.cc:2781] T 66f17c8828054dacb83c5f1343363424 P e85eb50b63bd4eb09824856f7241940a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:35:39.536659 2025 raft_consensus.cc:686] T 66f17c8828054dacb83c5f1343363424 P e85eb50b63bd4eb09824856f7241940a [term 1 LEADER]: Becoming Leader. State: Replica: e85eb50b63bd4eb09824856f7241940a, State: Running, Role: LEADER
I20260620 23:35:39.537230 2025 consensus_queue.cc:237] T 66f17c8828054dacb83c5f1343363424 P e85eb50b63bd4eb09824856f7241940a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebfb31c16f54e308a4bdc944630bd61" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42637 } } peers { permanent_uuid: "8ba7348efe9d4a87bb7e0cfa35140f5a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34037 } } peers { permanent_uuid: "e85eb50b63bd4eb09824856f7241940a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43707 } }
I20260620 23:35:39.542344 1515 catalog_manager.cc:5495] T 66f17c8828054dacb83c5f1343363424 P e85eb50b63bd4eb09824856f7241940a reported cstate change: term changed from 0 to 1, leader changed from <none> to e85eb50b63bd4eb09824856f7241940a (127.0.97.3). New cstate: current_term: 1 leader_uuid: "e85eb50b63bd4eb09824856f7241940a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebfb31c16f54e308a4bdc944630bd61" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ba7348efe9d4a87bb7e0cfa35140f5a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e85eb50b63bd4eb09824856f7241940a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43707 } health_report { overall_health: HEALTHY } } }
I20260620 23:35:39.658573 2033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:39.652242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46242 (local address 127.0.97.3:43707)
0620 23:35:39.652778 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:39.652784 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:39.652996 (+ 212us) server_negotiation.cc:407] Connection header received
0620 23:35:39.653342 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:39.653346 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:39.653429 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:39.653588 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:35:39.654049 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:39.654740 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:39.656215 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:39.656644 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:39.656807 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:39.657343 (+ 536us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:35:39.657889 (+ 546us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:35:39.658050 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:39.658186 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 23:35:39.658272 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":273,"threads_started":1}
I20260620 23:35:39.666694 1672 raft_consensus.cc:1261] T 66f17c8828054dacb83c5f1343363424 P 0ebfb31c16f54e308a4bdc944630bd61 [term 1 FOLLOWER]: Refusing update from remote peer e85eb50b63bd4eb09824856f7241940a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:35:39.666685 1816 raft_consensus.cc:1261] T 66f17c8828054dacb83c5f1343363424 P 8ba7348efe9d4a87bb7e0cfa35140f5a [term 1 FOLLOWER]: Refusing update from remote peer e85eb50b63bd4eb09824856f7241940a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:35:39.667737 2030 consensus_queue.cc:1035] T 66f17c8828054dacb83c5f1343363424 P e85eb50b63bd4eb09824856f7241940a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfb31c16f54e308a4bdc944630bd61" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:35:39.668097 2025 consensus_queue.cc:1035] T 66f17c8828054dacb83c5f1343363424 P e85eb50b63bd4eb09824856f7241940a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ba7348efe9d4a87bb7e0cfa35140f5a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:35:39.703210 1514 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:35:39.703517 1514 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:35:39.707085 1514 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b7bda5b0072b4a87b0297077f313def5: Sending DeleteTablet for 3 replicas of tablet 66f17c8828054dacb83c5f1343363424
I20260620 23:35:39.708171 1791 tablet_service.cc:1514] Processing DeleteTablet for tablet 66f17c8828054dacb83c5f1343363424 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:35:39 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53742
I20260620 23:35:39.708216 1652 tablet_service.cc:1514] Processing DeleteTablet for tablet 66f17c8828054dacb83c5f1343363424 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:35:39 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57654
I20260620 23:35:39.709213 2048 tablet_replica.cc:330] T 66f17c8828054dacb83c5f1343363424 P 0ebfb31c16f54e308a4bdc944630bd61: stopping tablet replica
I20260620 23:35:39.709374 1930 tablet_service.cc:1514] Processing DeleteTablet for tablet 66f17c8828054dacb83c5f1343363424 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:35:39 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46230
I20260620 23:35:39.709946 2048 raft_consensus.cc:2229] T 66f17c8828054dacb83c5f1343363424 P 0ebfb31c16f54e308a4bdc944630bd61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:35:39.710616 2048 raft_consensus.cc:2258] T 66f17c8828054dacb83c5f1343363424 P 0ebfb31c16f54e308a4bdc944630bd61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:35:39.711696 1514 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60390:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:35:39.712421 1514 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:35:39.714381 2047 tablet_replica.cc:330] T 66f17c8828054dacb83c5f1343363424 P 8ba7348efe9d4a87bb7e0cfa35140f5a: stopping tablet replica
I20260620 23:35:39.714782 2047 raft_consensus.cc:2229] T 66f17c8828054dacb83c5f1343363424 P 8ba7348efe9d4a87bb7e0cfa35140f5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:35:39.715241 2047 raft_consensus.cc:2258] T 66f17c8828054dacb83c5f1343363424 P 8ba7348efe9d4a87bb7e0cfa35140f5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:35:39.715773 2048 ts_tablet_manager.cc:1905] T 66f17c8828054dacb83c5f1343363424 P 0ebfb31c16f54e308a4bdc944630bd61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:35:39.722268 1929 tablet_service.cc:1467] Processing CreateTablet for tablet 17c55b43e18b409d811fd5caede5abfb (DEFAULT_TABLE table=test-table [id=655947f78bfe4c5e910ab5f3c919ba74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:35:39.722373 1791 tablet_service.cc:1467] Processing CreateTablet for tablet 17c55b43e18b409d811fd5caede5abfb (DEFAULT_TABLE table=test-table [id=655947f78bfe4c5e910ab5f3c919ba74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:35:39.722502 1652 tablet_service.cc:1467] Processing CreateTablet for tablet 17c55b43e18b409d811fd5caede5abfb (DEFAULT_TABLE table=test-table [id=655947f78bfe4c5e910ab5f3c919ba74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:35:39.722806 1791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17c55b43e18b409d811fd5caede5abfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:35:39.722894 1929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17c55b43e18b409d811fd5caede5abfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:35:39.722978 1652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17c55b43e18b409d811fd5caede5abfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:35:39.727873 2049 tablet_replica.cc:330] T 66f17c8828054dacb83c5f1343363424 P e85eb50b63bd4eb09824856f7241940a: stopping tablet replica
I20260620 23:35:39.728379 2049 raft_consensus.cc:2229] T 66f17c8828054dacb83c5f1343363424 P e85eb50b63bd4eb09824856f7241940a [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:35:39.729132 2049 raft_consensus.cc:2258] T 66f17c8828054dacb83c5f1343363424 P e85eb50b63bd4eb09824856f7241940a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:35:39.730384 2047 ts_tablet_manager.cc:1905] T 66f17c8828054dacb83c5f1343363424 P 8ba7348efe9d4a87bb7e0cfa35140f5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:35:39.745396 2049 ts_tablet_manager.cc:1905] T 66f17c8828054dacb83c5f1343363424 P e85eb50b63bd4eb09824856f7241940a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:35:39.753963 2048 ts_tablet_manager.cc:1918] T 66f17c8828054dacb83c5f1343363424 P 0ebfb31c16f54e308a4bdc944630bd61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:35:39.754208 2048 log.cc:1196] T 66f17c8828054dacb83c5f1343363424 P 0ebfb31c16f54e308a4bdc944630bd61: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/ts-0/wal/wals/66f17c8828054dacb83c5f1343363424
I20260620 23:35:39.758368 2051 tablet_bootstrap.cc:492] T 17c55b43e18b409d811fd5caede5abfb P 0ebfb31c16f54e308a4bdc944630bd61: Bootstrap starting.
I20260620 23:35:39.761399 2051 tablet_bootstrap.cc:654] T 17c55b43e18b409d811fd5caede5abfb P 0ebfb31c16f54e308a4bdc944630bd61: Neither blocks nor log segments found. Creating new log.
I20260620 23:35:39.763151 2052 tablet_bootstrap.cc:492] T 17c55b43e18b409d811fd5caede5abfb P 8ba7348efe9d4a87bb7e0cfa35140f5a: Bootstrap starting.
I20260620 23:35:39.764936 2047 ts_tablet_manager.cc:1918] T 66f17c8828054dacb83c5f1343363424 P 8ba7348efe9d4a87bb7e0cfa35140f5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:35:39.765137 2047 log.cc:1196] T 66f17c8828054dacb83c5f1343363424 P 8ba7348efe9d4a87bb7e0cfa35140f5a: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/ts-1/wal/wals/66f17c8828054dacb83c5f1343363424
I20260620 23:35:39.765635 2052 tablet_bootstrap.cc:654] T 17c55b43e18b409d811fd5caede5abfb P 8ba7348efe9d4a87bb7e0cfa35140f5a: Neither blocks nor log segments found. Creating new log.
I20260620 23:35:39.765673 2047 ts_tablet_manager.cc:1939] T 66f17c8828054dacb83c5f1343363424 P 8ba7348efe9d4a87bb7e0cfa35140f5a: Deleting consensus metadata
I20260620 23:35:39.754892 2048 ts_tablet_manager.cc:1939] T 66f17c8828054dacb83c5f1343363424 P 0ebfb31c16f54e308a4bdc944630bd61: Deleting consensus metadata
I20260620 23:35:39.770768 1500 catalog_manager.cc:4841] TS 0ebfb31c16f54e308a4bdc944630bd61 (127.0.97.1:42637): tablet 66f17c8828054dacb83c5f1343363424 (table test-table [id=e89b125b2b774d01b22a7c295ff522b6]) successfully deleted
I20260620 23:35:39.771699 2050 tablet_bootstrap.cc:492] T 17c55b43e18b409d811fd5caede5abfb P e85eb50b63bd4eb09824856f7241940a: Bootstrap starting.
I20260620 23:35:39.772548 1501 catalog_manager.cc:4841] TS 8ba7348efe9d4a87bb7e0cfa35140f5a (127.0.97.2:34037): tablet 66f17c8828054dacb83c5f1343363424 (table test-table [id=e89b125b2b774d01b22a7c295ff522b6]) successfully deleted
I20260620 23:35:39.773326 2051 tablet_bootstrap.cc:492] T 17c55b43e18b409d811fd5caede5abfb P 0ebfb31c16f54e308a4bdc944630bd61: No bootstrap required, opened a new log
I20260620 23:35:39.773488 2052 tablet_bootstrap.cc:492] T 17c55b43e18b409d811fd5caede5abfb P 8ba7348efe9d4a87bb7e0cfa35140f5a: No bootstrap required, opened a new log
I20260620 23:35:39.773486 2051 ts_tablet_manager.cc:1397] T 17c55b43e18b409d811fd5caede5abfb P 0ebfb31c16f54e308a4bdc944630bd61: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20260620 23:35:39.773728 2052 ts_tablet_manager.cc:1397] T 17c55b43e18b409d811fd5caede5abfb P 8ba7348efe9d4a87bb7e0cfa35140f5a: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.005s
I20260620 23:35:39.774286 2050 tablet_bootstrap.cc:654] T 17c55b43e18b409d811fd5caede5abfb P e85eb50b63bd4eb09824856f7241940a: Neither blocks nor log segments found. Creating new log.
I20260620 23:35:39.774323 2051 raft_consensus.cc:348] T 17c55b43e18b409d811fd5caede5abfb P 0ebfb31c16f54e308a4bdc944630bd61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebfb31c16f54e308a4bdc944630bd61" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42637 } } peers { permanent_uuid: "8ba7348efe9d4a87bb7e0cfa35140f5a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34037 } } peers { permanent_uuid: "e85eb50b63bd4eb09824856f7241940a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43707 } }
I20260620 23:35:39.774576 2051 raft_consensus.cc:374] T 17c55b43e18b409d811fd5caede5abfb P 0ebfb31c16f54e308a4bdc944630bd61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:35:39.774682 2051 raft_consensus.cc:729] T 17c55b43e18b409d811fd5caede5abfb P 0ebfb31c16f54e308a4bdc944630bd61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ebfb31c16f54e308a4bdc944630bd61, State: Initialized, Role: FOLLOWER
I20260620 23:35:39.774782 2049 ts_tablet_manager.cc:1918] T 66f17c8828054dacb83c5f1343363424 P e85eb50b63bd4eb09824856f7241940a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:35:39.774703 2052 raft_consensus.cc:348] T 17c55b43e18b409d811fd5caede5abfb P 8ba7348efe9d4a87bb7e0cfa35140f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebfb31c16f54e308a4bdc944630bd61" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42637 } } peers { permanent_uuid: "8ba7348efe9d4a87bb7e0cfa35140f5a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34037 } } peers { permanent_uuid: "e85eb50b63bd4eb09824856f7241940a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43707 } }
I20260620 23:35:39.775104 2052 raft_consensus.cc:374] T 17c55b43e18b409d811fd5caede5abfb P 8ba7348efe9d4a87bb7e0cfa35140f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:35:39.774974 2051 consensus_queue.cc:260] T 17c55b43e18b409d811fd5caede5abfb P 0ebfb31c16f54e308a4bdc944630bd61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebfb31c16f54e308a4bdc944630bd61" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42637 } } peers { permanent_uuid: "8ba7348efe9d4a87bb7e0cfa35140f5a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34037 } } peers { permanent_uuid: "e85eb50b63bd4eb09824856f7241940a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43707 } }
I20260620 23:35:39.775293 2052 raft_consensus.cc:729] T 17c55b43e18b409d811fd5caede5abfb P 8ba7348efe9d4a87bb7e0cfa35140f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ba7348efe9d4a87bb7e0cfa35140f5a, State: Initialized, Role: FOLLOWER
I20260620 23:35:39.775566 2052 consensus_queue.cc:260] T 17c55b43e18b409d811fd5caede5abfb P 8ba7348efe9d4a87bb7e0cfa35140f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebfb31c16f54e308a4bdc944630bd61" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42637 } } peers { permanent_uuid: "8ba7348efe9d4a87bb7e0cfa35140f5a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34037 } } peers { permanent_uuid: "e85eb50b63bd4eb09824856f7241940a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43707 } }
I20260620 23:35:39.776741 2052 ts_tablet_manager.cc:1428] T 17c55b43e18b409d811fd5caede5abfb P 8ba7348efe9d4a87bb7e0cfa35140f5a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:35:39.779044 2049 log.cc:1196] T 66f17c8828054dacb83c5f1343363424 P e85eb50b63bd4eb09824856f7241940a: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/ts-2/wal/wals/66f17c8828054dacb83c5f1343363424
I20260620 23:35:39.779811 2049 ts_tablet_manager.cc:1939] T 66f17c8828054dacb83c5f1343363424 P e85eb50b63bd4eb09824856f7241940a: Deleting consensus metadata
I20260620 23:35:39.781803 1502 catalog_manager.cc:4841] TS e85eb50b63bd4eb09824856f7241940a (127.0.97.3:43707): tablet 66f17c8828054dacb83c5f1343363424 (table test-table [id=e89b125b2b774d01b22a7c295ff522b6]) successfully deleted
I20260620 23:35:39.782285 2051 ts_tablet_manager.cc:1428] T 17c55b43e18b409d811fd5caede5abfb P 0ebfb31c16f54e308a4bdc944630bd61: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20260620 23:35:39.783950 2050 tablet_bootstrap.cc:492] T 17c55b43e18b409d811fd5caede5abfb P e85eb50b63bd4eb09824856f7241940a: No bootstrap required, opened a new log
I20260620 23:35:39.784126 2050 ts_tablet_manager.cc:1397] T 17c55b43e18b409d811fd5caede5abfb P e85eb50b63bd4eb09824856f7241940a: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20260620 23:35:39.784850 2050 raft_consensus.cc:348] T 17c55b43e18b409d811fd5caede5abfb P e85eb50b63bd4eb09824856f7241940a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebfb31c16f54e308a4bdc944630bd61" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42637 } } peers { permanent_uuid: "8ba7348efe9d4a87bb7e0cfa35140f5a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34037 } } peers { permanent_uuid: "e85eb50b63bd4eb09824856f7241940a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43707 } }
I20260620 23:35:39.785120 2050 raft_consensus.cc:374] T 17c55b43e18b409d811fd5caede5abfb P e85eb50b63bd4eb09824856f7241940a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:35:39.785233 2050 raft_consensus.cc:729] T 17c55b43e18b409d811fd5caede5abfb P e85eb50b63bd4eb09824856f7241940a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e85eb50b63bd4eb09824856f7241940a, State: Initialized, Role: FOLLOWER
I20260620 23:35:39.785508 2050 consensus_queue.cc:260] T 17c55b43e18b409d811fd5caede5abfb P e85eb50b63bd4eb09824856f7241940a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebfb31c16f54e308a4bdc944630bd61" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42637 } } peers { permanent_uuid: "8ba7348efe9d4a87bb7e0cfa35140f5a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34037 } } peers { permanent_uuid: "e85eb50b63bd4eb09824856f7241940a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43707 } }
I20260620 23:35:39.786029 2050 ts_tablet_manager.cc:1428] T 17c55b43e18b409d811fd5caede5abfb P e85eb50b63bd4eb09824856f7241940a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260620 23:35:39.787766 2044 raft_consensus.cc:482] T 17c55b43e18b409d811fd5caede5abfb P e85eb50b63bd4eb09824856f7241940a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:35:39.787976 2044 raft_consensus.cc:504] T 17c55b43e18b409d811fd5caede5abfb P e85eb50b63bd4eb09824856f7241940a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebfb31c16f54e308a4bdc944630bd61" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42637 } } peers { permanent_uuid: "8ba7348efe9d4a87bb7e0cfa35140f5a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34037 } } peers { permanent_uuid: "e85eb50b63bd4eb09824856f7241940a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43707 } }
I20260620 23:35:39.788847 2044 leader_election.cc:288] T 17c55b43e18b409d811fd5caede5abfb P e85eb50b63bd4eb09824856f7241940a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ebfb31c16f54e308a4bdc944630bd61 (127.0.97.1:42637), 8ba7348efe9d4a87bb7e0cfa35140f5a (127.0.97.2:34037)
I20260620 23:35:39.789358 1816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17c55b43e18b409d811fd5caede5abfb" candidate_uuid: "e85eb50b63bd4eb09824856f7241940a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ba7348efe9d4a87bb7e0cfa35140f5a" is_pre_election: true
I20260620 23:35:39.789579 1672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17c55b43e18b409d811fd5caede5abfb" candidate_uuid: "e85eb50b63bd4eb09824856f7241940a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ebfb31c16f54e308a4bdc944630bd61" is_pre_election: true
I20260620 23:35:39.789641 1816 raft_consensus.cc:2454] T 17c55b43e18b409d811fd5caede5abfb P 8ba7348efe9d4a87bb7e0cfa35140f5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e85eb50b63bd4eb09824856f7241940a in term 0.
I20260620 23:35:39.789821 1672 raft_consensus.cc:2454] T 17c55b43e18b409d811fd5caede5abfb P 0ebfb31c16f54e308a4bdc944630bd61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e85eb50b63bd4eb09824856f7241940a in term 0.
I20260620 23:35:39.790287 1882 leader_election.cc:302] T 17c55b43e18b409d811fd5caede5abfb P e85eb50b63bd4eb09824856f7241940a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfb31c16f54e308a4bdc944630bd61, e85eb50b63bd4eb09824856f7241940a; no voters:
I20260620 23:35:39.790627 2044 raft_consensus.cc:2781] T 17c55b43e18b409d811fd5caede5abfb P e85eb50b63bd4eb09824856f7241940a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:35:39.790752 2044 raft_consensus.cc:482] T 17c55b43e18b409d811fd5caede5abfb P e85eb50b63bd4eb09824856f7241940a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:35:39.790853 2044 raft_consensus.cc:3037] T 17c55b43e18b409d811fd5caede5abfb P e85eb50b63bd4eb09824856f7241940a [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:35:39.795156 2044 raft_consensus.cc:504] T 17c55b43e18b409d811fd5caede5abfb P e85eb50b63bd4eb09824856f7241940a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebfb31c16f54e308a4bdc944630bd61" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42637 } } peers { permanent_uuid: "8ba7348efe9d4a87bb7e0cfa35140f5a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34037 } } peers { permanent_uuid: "e85eb50b63bd4eb09824856f7241940a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43707 } }
I20260620 23:35:39.796011 2044 leader_election.cc:288] T 17c55b43e18b409d811fd5caede5abfb P e85eb50b63bd4eb09824856f7241940a [CANDIDATE]: Term 1 election: Requested vote from peers 0ebfb31c16f54e308a4bdc944630bd61 (127.0.97.1:42637), 8ba7348efe9d4a87bb7e0cfa35140f5a (127.0.97.2:34037)
I20260620 23:35:39.796413 1816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17c55b43e18b409d811fd5caede5abfb" candidate_uuid: "e85eb50b63bd4eb09824856f7241940a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ba7348efe9d4a87bb7e0cfa35140f5a"
I20260620 23:35:39.796679 1816 raft_consensus.cc:3037] T 17c55b43e18b409d811fd5caede5abfb P 8ba7348efe9d4a87bb7e0cfa35140f5a [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:35:39.796680 1672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17c55b43e18b409d811fd5caede5abfb" candidate_uuid: "e85eb50b63bd4eb09824856f7241940a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ebfb31c16f54e308a4bdc944630bd61"
I20260620 23:35:39.796891 1672 raft_consensus.cc:3037] T 17c55b43e18b409d811fd5caede5abfb P 0ebfb31c16f54e308a4bdc944630bd61 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:35:39.798766 1816 raft_consensus.cc:2454] T 17c55b43e18b409d811fd5caede5abfb P 8ba7348efe9d4a87bb7e0cfa35140f5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e85eb50b63bd4eb09824856f7241940a in term 1.
I20260620 23:35:39.799257 1883 leader_election.cc:302] T 17c55b43e18b409d811fd5caede5abfb P e85eb50b63bd4eb09824856f7241940a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ba7348efe9d4a87bb7e0cfa35140f5a, e85eb50b63bd4eb09824856f7241940a; no voters:
I20260620 23:35:39.799813 1672 raft_consensus.cc:2454] T 17c55b43e18b409d811fd5caede5abfb P 0ebfb31c16f54e308a4bdc944630bd61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e85eb50b63bd4eb09824856f7241940a in term 1.
I20260620 23:35:39.800156 2044 raft_consensus.cc:2781] T 17c55b43e18b409d811fd5caede5abfb P e85eb50b63bd4eb09824856f7241940a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:35:39.800335 2044 raft_consensus.cc:686] T 17c55b43e18b409d811fd5caede5abfb P e85eb50b63bd4eb09824856f7241940a [term 1 LEADER]: Becoming Leader. State: Replica: e85eb50b63bd4eb09824856f7241940a, State: Running, Role: LEADER
I20260620 23:35:39.800894 2044 consensus_queue.cc:237] T 17c55b43e18b409d811fd5caede5abfb P e85eb50b63bd4eb09824856f7241940a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebfb31c16f54e308a4bdc944630bd61" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42637 } } peers { permanent_uuid: "8ba7348efe9d4a87bb7e0cfa35140f5a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34037 } } peers { permanent_uuid: "e85eb50b63bd4eb09824856f7241940a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43707 } }
I20260620 23:35:39.803714 1514 catalog_manager.cc:5495] T 17c55b43e18b409d811fd5caede5abfb P e85eb50b63bd4eb09824856f7241940a reported cstate change: term changed from 0 to 1, leader changed from <none> to e85eb50b63bd4eb09824856f7241940a (127.0.97.3). New cstate: current_term: 1 leader_uuid: "e85eb50b63bd4eb09824856f7241940a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebfb31c16f54e308a4bdc944630bd61" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ba7348efe9d4a87bb7e0cfa35140f5a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e85eb50b63bd4eb09824856f7241940a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43707 } health_report { overall_health: HEALTHY } } }
Jun 20 23:35:39 dist-test-slave-tcgx krb5kdc[1470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:35:39.848410 2063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:39.833457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44817 (local address 127.0.0.1:60400)
0620 23:35:39.834240 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0620 23:35:39.834256 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:35:39.834478 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:35:39.834802 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:35:39.834807 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:35:39.835571 (+ 764us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:35:39.835786 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:39.835796 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:39.836831 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:39.836837 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:39.838211 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:39.838223 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:39.838968 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:39.838974 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:39.839137 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:39.839825 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:35:39.839841 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:35:39.842344 (+ 2503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:35:39.845450 (+ 3106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:35:39.845464 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:35:39.845472 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:35:39.846071 (+ 599us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:35:39.846425 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:35:39.846430 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:35:39.846434 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:35:39.846604 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:35:39.847172 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:35:39.847181 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:35:39.847386 (+ 205us) client_negotiation.cc:769] Sending connection context
0620 23:35:39.847589 (+ 203us) client_negotiation.cc:240] Negotiation successful
0620 23:35:39.847938 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"thread_start_us":533,"threads_started":1}
I20260620 23:35:39.848548 2062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:39.833413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60400 (local address 127.0.97.62:44817)
0620 23:35:39.834054 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:39.834062 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:39.834371 (+ 309us) server_negotiation.cc:407] Connection header received
0620 23:35:39.834604 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:39.834615 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:39.834665 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:39.834774 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:35:39.835940 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:39.836687 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:39.838349 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:39.838774 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:39.838912 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:39.842561 (+ 3649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:35:39.842566 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:35:39.842569 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:35:39.842598 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:35:39.845311 (+ 2713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:39.846195 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:39.846199 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:39.846201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:39.846318 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:39.846722 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:39.846725 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:39.846727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:39.847051 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:35:39.847172 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:39.847704 (+ 532us) server_negotiation.cc:299] Negotiation successful
0620 23:35:39.848122 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":177,"server-negotiator.queue_time_us":477,"thread_start_us":308,"threads_started":1}
I20260620 23:35:39.854220 1514 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60400:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 23:35:39.855584 1514 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:35:39.863813 1929 tablet_service.cc:1467] Processing CreateTablet for tablet 231deff76c3f44e683146ae7e15766a5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7bf77cc22173445180060ac13f0d2c7d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 23:35:39.863793 1791 tablet_service.cc:1467] Processing CreateTablet for tablet 231deff76c3f44e683146ae7e15766a5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7bf77cc22173445180060ac13f0d2c7d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 23:35:39.863930 1652 tablet_service.cc:1467] Processing CreateTablet for tablet 231deff76c3f44e683146ae7e15766a5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7bf77cc22173445180060ac13f0d2c7d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 23:35:39.864395 1791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 231deff76c3f44e683146ae7e15766a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:35:39.864387 1929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 231deff76c3f44e683146ae7e15766a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:35:39.864463 1652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 231deff76c3f44e683146ae7e15766a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:35:39.871932 2050 tablet_bootstrap.cc:492] T 231deff76c3f44e683146ae7e15766a5 P e85eb50b63bd4eb09824856f7241940a: Bootstrap starting.
I20260620 23:35:39.871932 2052 tablet_bootstrap.cc:492] T 231deff76c3f44e683146ae7e15766a5 P 8ba7348efe9d4a87bb7e0cfa35140f5a: Bootstrap starting.
I20260620 23:35:39.873703 2052 tablet_bootstrap.cc:654] T 231deff76c3f44e683146ae7e15766a5 P 8ba7348efe9d4a87bb7e0cfa35140f5a: Neither blocks nor log segments found. Creating new log.
I20260620 23:35:39.873715 2050 tablet_bootstrap.cc:654] T 231deff76c3f44e683146ae7e15766a5 P e85eb50b63bd4eb09824856f7241940a: Neither blocks nor log segments found. Creating new log.
I20260620 23:35:39.874841 2051 tablet_bootstrap.cc:492] T 231deff76c3f44e683146ae7e15766a5 P 0ebfb31c16f54e308a4bdc944630bd61: Bootstrap starting.
I20260620 23:35:39.877151 2051 tablet_bootstrap.cc:654] T 231deff76c3f44e683146ae7e15766a5 P 0ebfb31c16f54e308a4bdc944630bd61: Neither blocks nor log segments found. Creating new log.
I20260620 23:35:39.889232 2051 tablet_bootstrap.cc:492] T 231deff76c3f44e683146ae7e15766a5 P 0ebfb31c16f54e308a4bdc944630bd61: No bootstrap required, opened a new log
I20260620 23:35:39.889541 2051 ts_tablet_manager.cc:1397] T 231deff76c3f44e683146ae7e15766a5 P 0ebfb31c16f54e308a4bdc944630bd61: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.010s
I20260620 23:35:39.889703 2052 tablet_bootstrap.cc:492] T 231deff76c3f44e683146ae7e15766a5 P 8ba7348efe9d4a87bb7e0cfa35140f5a: No bootstrap required, opened a new log
I20260620 23:35:39.889910 2052 ts_tablet_manager.cc:1397] T 231deff76c3f44e683146ae7e15766a5 P 8ba7348efe9d4a87bb7e0cfa35140f5a: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.006s
I20260620 23:35:39.890709 2052 raft_consensus.cc:348] T 231deff76c3f44e683146ae7e15766a5 P 8ba7348efe9d4a87bb7e0cfa35140f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebfb31c16f54e308a4bdc944630bd61" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42637 } } peers { permanent_uuid: "8ba7348efe9d4a87bb7e0cfa35140f5a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34037 } } peers { permanent_uuid: "e85eb50b63bd4eb09824856f7241940a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43707 } }
I20260620 23:35:39.891054 2052 raft_consensus.cc:374] T 231deff76c3f44e683146ae7e15766a5 P 8ba7348efe9d4a87bb7e0cfa35140f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:35:39.890727 2051 raft_consensus.cc:348] T 231deff76c3f44e683146ae7e15766a5 P 0ebfb31c16f54e308a4bdc944630bd61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebfb31c16f54e308a4bdc944630bd61" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42637 } } peers { permanent_uuid: "8ba7348efe9d4a87bb7e0cfa35140f5a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34037 } } peers { permanent_uuid: "e85eb50b63bd4eb09824856f7241940a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43707 } }
I20260620 23:35:39.891180 2052 raft_consensus.cc:729] T 231deff76c3f44e683146ae7e15766a5 P 8ba7348efe9d4a87bb7e0cfa35140f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ba7348efe9d4a87bb7e0cfa35140f5a, State: Initialized, Role: FOLLOWER
I20260620 23:35:39.891250 2051 raft_consensus.cc:374] T 231deff76c3f44e683146ae7e15766a5 P 0ebfb31c16f54e308a4bdc944630bd61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:35:39.891566 2051 raft_consensus.cc:729] T 231deff76c3f44e683146ae7e15766a5 P 0ebfb31c16f54e308a4bdc944630bd61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ebfb31c16f54e308a4bdc944630bd61, State: Initialized, Role: FOLLOWER
I20260620 23:35:39.891466 2052 consensus_queue.cc:260] T 231deff76c3f44e683146ae7e15766a5 P 8ba7348efe9d4a87bb7e0cfa35140f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebfb31c16f54e308a4bdc944630bd61" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42637 } } peers { permanent_uuid: "8ba7348efe9d4a87bb7e0cfa35140f5a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34037 } } peers { permanent_uuid: "e85eb50b63bd4eb09824856f7241940a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43707 } }
I20260620 23:35:39.891999 2051 consensus_queue.cc:260] T 231deff76c3f44e683146ae7e15766a5 P 0ebfb31c16f54e308a4bdc944630bd61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebfb31c16f54e308a4bdc944630bd61" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42637 } } peers { permanent_uuid: "8ba7348efe9d4a87bb7e0cfa35140f5a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34037 } } peers { permanent_uuid: "e85eb50b63bd4eb09824856f7241940a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43707 } }
I20260620 23:35:39.892161 2052 ts_tablet_manager.cc:1428] T 231deff76c3f44e683146ae7e15766a5 P 8ba7348efe9d4a87bb7e0cfa35140f5a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 23:35:39.892053 2036 tablet_replica.cc:439] T 231deff76c3f44e683146ae7e15766a5 P 8ba7348efe9d4a87bb7e0cfa35140f5a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebfb31c16f54e308a4bdc944630bd61" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42637 } } peers { permanent_uuid: "8ba7348efe9d4a87bb7e0cfa35140f5a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34037 } } peers { permanent_uuid: "e85eb50b63bd4eb09824856f7241940a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43707 } } }
I20260620 23:35:39.892418 2036 tablet_replica.cc:442] T 231deff76c3f44e683146ae7e15766a5 P 8ba7348efe9d4a87bb7e0cfa35140f5a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:35:39.892776 2037 tablet_replica.cc:439] T 231deff76c3f44e683146ae7e15766a5 P 0ebfb31c16f54e308a4bdc944630bd61: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebfb31c16f54e308a4bdc944630bd61" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42637 } } peers { permanent_uuid: "8ba7348efe9d4a87bb7e0cfa35140f5a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34037 } } peers { permanent_uuid: "e85eb50b63bd4eb09824856f7241940a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43707 } } }
I20260620 23:35:39.893127 2037 tablet_replica.cc:442] T 231deff76c3f44e683146ae7e15766a5 P 0ebfb31c16f54e308a4bdc944630bd61: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:35:39.894717 2050 tablet_bootstrap.cc:492] T 231deff76c3f44e683146ae7e15766a5 P e85eb50b63bd4eb09824856f7241940a: No bootstrap required, opened a new log
I20260620 23:35:39.894965 2050 ts_tablet_manager.cc:1397] T 231deff76c3f44e683146ae7e15766a5 P e85eb50b63bd4eb09824856f7241940a: Time spent bootstrapping tablet: real 0.023s user 0.020s sys 0.000s
I20260620 23:35:39.895328 2037 raft_consensus.cc:482] T 231deff76c3f44e683146ae7e15766a5 P 0ebfb31c16f54e308a4bdc944630bd61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:35:39.895550 2037 raft_consensus.cc:504] T 231deff76c3f44e683146ae7e15766a5 P 0ebfb31c16f54e308a4bdc944630bd61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebfb31c16f54e308a4bdc944630bd61" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42637 } } peers { permanent_uuid: "8ba7348efe9d4a87bb7e0cfa35140f5a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34037 } } peers { permanent_uuid: "e85eb50b63bd4eb09824856f7241940a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43707 } }
I20260620 23:35:39.895853 2050 raft_consensus.cc:348] T 231deff76c3f44e683146ae7e15766a5 P e85eb50b63bd4eb09824856f7241940a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebfb31c16f54e308a4bdc944630bd61" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42637 } } peers { permanent_uuid: "8ba7348efe9d4a87bb7e0cfa35140f5a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34037 } } peers { permanent_uuid: "e85eb50b63bd4eb09824856f7241940a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43707 } }
I20260620 23:35:39.896183 2050 raft_consensus.cc:374] T 231deff76c3f44e683146ae7e15766a5 P e85eb50b63bd4eb09824856f7241940a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:35:39.896292 2050 raft_consensus.cc:729] T 231deff76c3f44e683146ae7e15766a5 P e85eb50b63bd4eb09824856f7241940a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e85eb50b63bd4eb09824856f7241940a, State: Initialized, Role: FOLLOWER
I20260620 23:35:39.896553 2050 consensus_queue.cc:260] T 231deff76c3f44e683146ae7e15766a5 P e85eb50b63bd4eb09824856f7241940a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebfb31c16f54e308a4bdc944630bd61" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42637 } } peers { permanent_uuid: "8ba7348efe9d4a87bb7e0cfa35140f5a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34037 } } peers { permanent_uuid: "e85eb50b63bd4eb09824856f7241940a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43707 } }
I20260620 23:35:39.897147 2050 ts_tablet_manager.cc:1428] T 231deff76c3f44e683146ae7e15766a5 P e85eb50b63bd4eb09824856f7241940a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:35:39.897317 2037 leader_election.cc:288] T 231deff76c3f44e683146ae7e15766a5 P 0ebfb31c16f54e308a4bdc944630bd61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ba7348efe9d4a87bb7e0cfa35140f5a (127.0.97.2:34037), e85eb50b63bd4eb09824856f7241940a (127.0.97.3:43707)
I20260620 23:35:39.897037 2044 tablet_replica.cc:439] T 231deff76c3f44e683146ae7e15766a5 P e85eb50b63bd4eb09824856f7241940a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebfb31c16f54e308a4bdc944630bd61" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42637 } } peers { permanent_uuid: "8ba7348efe9d4a87bb7e0cfa35140f5a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34037 } } peers { permanent_uuid: "e85eb50b63bd4eb09824856f7241940a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43707 } } }
I20260620 23:35:39.897840 2044 tablet_replica.cc:442] T 231deff76c3f44e683146ae7e15766a5 P e85eb50b63bd4eb09824856f7241940a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:35:39.902072 2051 ts_tablet_manager.cc:1428] T 231deff76c3f44e683146ae7e15766a5 P 0ebfb31c16f54e308a4bdc944630bd61: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20260620 23:35:39.908115 2033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:39.898058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50529 (local address 127.0.97.3:43707)
0620 23:35:39.898263 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:39.898269 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:39.898446 (+ 177us) server_negotiation.cc:407] Connection header received
0620 23:35:39.898721 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:39.898728 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:39.898800 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:39.898861 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:35:39.899778 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:39.900904 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:39.905560 (+ 4656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:39.906985 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:39.907131 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:39.907231 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:39.907654 (+ 423us) server_negotiation.cc:299] Negotiation successful
0620 23:35:39.907812 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 23:35:39.908895 2067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:39.897780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43707 (local address 127.0.97.1:50529)
0620 23:35:39.898346 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0620 23:35:39.898370 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 23:35:39.898558 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:35:39.899108 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:35:39.899113 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:35:39.899140 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:35:39.899563 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:39.899576 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:39.902420 (+ 2844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:39.902426 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:39.905390 (+ 2964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:39.905406 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:39.907204 (+ 1798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:39.907212 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:39.907411 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:39.907487 (+ 76us) client_negotiation.cc:769] Sending connection context
0620 23:35:39.908328 (+ 841us) client_negotiation.cc:240] Negotiation successful
0620 23:35:39.908481 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"mutex_wait_us":27,"thread_start_us":955,"threads_started":1}
I20260620 23:35:39.909874 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:39.898693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42939 (local address 127.0.97.2:34037)
0620 23:35:39.898865 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:39.898871 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:39.901989 (+ 3118us) server_negotiation.cc:407] Connection header received
0620 23:35:39.902297 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:39.902302 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:39.902382 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:39.902431 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:35:39.904226 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:39.905148 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:39.907629 (+ 2481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:39.908544 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:39.908689 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:39.908792 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:39.909428 (+ 636us) server_negotiation.cc:299] Negotiation successful
0620 23:35:39.909593 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 23:35:39.909755 1955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "231deff76c3f44e683146ae7e15766a5" candidate_uuid: "0ebfb31c16f54e308a4bdc944630bd61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e85eb50b63bd4eb09824856f7241940a" is_pre_election: true
I20260620 23:35:39.910267 1955 raft_consensus.cc:2454] T 231deff76c3f44e683146ae7e15766a5 P e85eb50b63bd4eb09824856f7241940a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfb31c16f54e308a4bdc944630bd61 in term 0.
I20260620 23:35:39.910833 2068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:39.898586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34037 (local address 127.0.97.1:42939)
0620 23:35:39.901878 (+ 3292us) negotiation.cc:107] Waiting for socket to connect
0620 23:35:39.901909 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 23:35:39.902096 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:35:39.903026 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:35:39.903033 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:35:39.903061 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:35:39.903966 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:39.903984 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:39.905350 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:39.905353 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:39.907463 (+ 2110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:39.907485 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:39.908776 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:39.908782 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:39.909234 (+ 452us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:39.909267 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 23:35:39.910274 (+ 1007us) client_negotiation.cc:240] Negotiation successful
0620 23:35:39.910439 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3164,"mutex_wait_us":21,"thread_start_us":3015,"threads_started":1}
I20260620 23:35:39.911031 1605 leader_election.cc:302] T 231deff76c3f44e683146ae7e15766a5 P 0ebfb31c16f54e308a4bdc944630bd61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfb31c16f54e308a4bdc944630bd61, e85eb50b63bd4eb09824856f7241940a; no voters:
I20260620 23:35:39.911495 2037 raft_consensus.cc:2781] T 231deff76c3f44e683146ae7e15766a5 P 0ebfb31c16f54e308a4bdc944630bd61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:35:39.911638 2037 raft_consensus.cc:482] T 231deff76c3f44e683146ae7e15766a5 P 0ebfb31c16f54e308a4bdc944630bd61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:35:39.911741 2037 raft_consensus.cc:3037] T 231deff76c3f44e683146ae7e15766a5 P 0ebfb31c16f54e308a4bdc944630bd61 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:35:39.915510 1816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "231deff76c3f44e683146ae7e15766a5" candidate_uuid: "0ebfb31c16f54e308a4bdc944630bd61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ba7348efe9d4a87bb7e0cfa35140f5a" is_pre_election: true
I20260620 23:35:39.915836 1816 raft_consensus.cc:2454] T 231deff76c3f44e683146ae7e15766a5 P 8ba7348efe9d4a87bb7e0cfa35140f5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfb31c16f54e308a4bdc944630bd61 in term 0.
I20260620 23:35:39.915769 2037 raft_consensus.cc:504] T 231deff76c3f44e683146ae7e15766a5 P 0ebfb31c16f54e308a4bdc944630bd61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebfb31c16f54e308a4bdc944630bd61" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42637 } } peers { permanent_uuid: "8ba7348efe9d4a87bb7e0cfa35140f5a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34037 } } peers { permanent_uuid: "e85eb50b63bd4eb09824856f7241940a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43707 } }
I20260620 23:35:39.916529 2037 leader_election.cc:288] T 231deff76c3f44e683146ae7e15766a5 P 0ebfb31c16f54e308a4bdc944630bd61 [CANDIDATE]: Term 1 election: Requested vote from peers 8ba7348efe9d4a87bb7e0cfa35140f5a (127.0.97.2:34037), e85eb50b63bd4eb09824856f7241940a (127.0.97.3:43707)
I20260620 23:35:39.917117 1955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "231deff76c3f44e683146ae7e15766a5" candidate_uuid: "0ebfb31c16f54e308a4bdc944630bd61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e85eb50b63bd4eb09824856f7241940a"
I20260620 23:35:39.917171 1816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "231deff76c3f44e683146ae7e15766a5" candidate_uuid: "0ebfb31c16f54e308a4bdc944630bd61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ba7348efe9d4a87bb7e0cfa35140f5a"
I20260620 23:35:39.917364 1955 raft_consensus.cc:3037] T 231deff76c3f44e683146ae7e15766a5 P e85eb50b63bd4eb09824856f7241940a [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:35:39.917378 1816 raft_consensus.cc:3037] T 231deff76c3f44e683146ae7e15766a5 P 8ba7348efe9d4a87bb7e0cfa35140f5a [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:35:39.920328 1816 raft_consensus.cc:2454] T 231deff76c3f44e683146ae7e15766a5 P 8ba7348efe9d4a87bb7e0cfa35140f5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfb31c16f54e308a4bdc944630bd61 in term 1.
I20260620 23:35:39.920632 1955 raft_consensus.cc:2454] T 231deff76c3f44e683146ae7e15766a5 P e85eb50b63bd4eb09824856f7241940a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfb31c16f54e308a4bdc944630bd61 in term 1.
I20260620 23:35:39.921088 1605 leader_election.cc:302] T 231deff76c3f44e683146ae7e15766a5 P 0ebfb31c16f54e308a4bdc944630bd61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfb31c16f54e308a4bdc944630bd61, e85eb50b63bd4eb09824856f7241940a; no voters:
I20260620 23:35:39.921424 2037 raft_consensus.cc:2781] T 231deff76c3f44e683146ae7e15766a5 P 0ebfb31c16f54e308a4bdc944630bd61 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:35:39.921967 2037 raft_consensus.cc:686] T 231deff76c3f44e683146ae7e15766a5 P 0ebfb31c16f54e308a4bdc944630bd61 [term 1 LEADER]: Becoming Leader. State: Replica: 0ebfb31c16f54e308a4bdc944630bd61, State: Running, Role: LEADER
I20260620 23:35:39.922681 2037 consensus_queue.cc:237] T 231deff76c3f44e683146ae7e15766a5 P 0ebfb31c16f54e308a4bdc944630bd61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebfb31c16f54e308a4bdc944630bd61" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42637 } } peers { permanent_uuid: "8ba7348efe9d4a87bb7e0cfa35140f5a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34037 } } peers { permanent_uuid: "e85eb50b63bd4eb09824856f7241940a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43707 } }
I20260620 23:35:39.925575 2069 tablet_replica.cc:439] T 231deff76c3f44e683146ae7e15766a5 P 0ebfb31c16f54e308a4bdc944630bd61: TxnStatusTablet state changed. Reason: New leader 0ebfb31c16f54e308a4bdc944630bd61. Latest consensus state: current_term: 1 leader_uuid: "0ebfb31c16f54e308a4bdc944630bd61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebfb31c16f54e308a4bdc944630bd61" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42637 } } peers { permanent_uuid: "8ba7348efe9d4a87bb7e0cfa35140f5a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34037 } } peers { permanent_uuid: "e85eb50b63bd4eb09824856f7241940a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43707 } } }
I20260620 23:35:39.925932 2069 tablet_replica.cc:442] T 231deff76c3f44e683146ae7e15766a5 P 0ebfb31c16f54e308a4bdc944630bd61: This TxnStatusTablet replica's current role is: LEADER
I20260620 23:35:39.927399 2071 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 23:35:39.927690 2071 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 23:35:39.927415 1514 catalog_manager.cc:5495] T 231deff76c3f44e683146ae7e15766a5 P 0ebfb31c16f54e308a4bdc944630bd61 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ebfb31c16f54e308a4bdc944630bd61 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "0ebfb31c16f54e308a4bdc944630bd61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebfb31c16f54e308a4bdc944630bd61" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42637 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ba7348efe9d4a87bb7e0cfa35140f5a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e85eb50b63bd4eb09824856f7241940a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43707 } health_report { overall_health: UNKNOWN } } }
I20260620 23:35:39.953300 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:39.945921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57668 (local address 127.0.97.1:42637)
0620 23:35:39.946089 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:39.946099 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:39.946119 (+ 20us) server_negotiation.cc:407] Connection header received
0620 23:35:39.946209 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:39.946214 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:39.946321 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:39.946481 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:35:39.946885 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:39.948509 (+ 1624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:39.950463 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:39.951184 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:39.951313 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:39.951835 (+ 522us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:35:39.952586 (+ 751us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:35:39.952730 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:39.952873 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 23:35:39.952956 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 23:35:39.953326 2063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:39.945390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42637 (local address 127.0.0.1:57668)
0620 23:35:39.945656 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 23:35:39.945678 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 23:35:39.945827 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:35:39.946510 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:35:39.946514 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:35:39.946527 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:35:39.946729 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:39.946740 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:39.948660 (+ 1920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:39.948666 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:39.950303 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:39.950319 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:39.951348 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:39.951355 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:39.951554 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:39.951629 (+ 75us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:35:39.952709 (+ 1080us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:35:39.952732 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 23:35:39.952827 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 23:35:39.952991 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260620 23:35:39.958475 1816 raft_consensus.cc:1261] T 231deff76c3f44e683146ae7e15766a5 P 8ba7348efe9d4a87bb7e0cfa35140f5a [term 1 FOLLOWER]: Refusing update from remote peer 0ebfb31c16f54e308a4bdc944630bd61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:35:39.958726 1955 raft_consensus.cc:1261] T 231deff76c3f44e683146ae7e15766a5 P e85eb50b63bd4eb09824856f7241940a [term 1 FOLLOWER]: Refusing update from remote peer 0ebfb31c16f54e308a4bdc944630bd61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:35:39.959219 2037 consensus_queue.cc:1035] T 231deff76c3f44e683146ae7e15766a5 P 0ebfb31c16f54e308a4bdc944630bd61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ba7348efe9d4a87bb7e0cfa35140f5a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:35:39.960271 2037 consensus_queue.cc:1035] T 231deff76c3f44e683146ae7e15766a5 P 0ebfb31c16f54e308a4bdc944630bd61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e85eb50b63bd4eb09824856f7241940a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:35:39.982331 2036 tablet_replica.cc:439] T 231deff76c3f44e683146ae7e15766a5 P 8ba7348efe9d4a87bb7e0cfa35140f5a: TxnStatusTablet state changed. Reason: New leader 0ebfb31c16f54e308a4bdc944630bd61. Latest consensus state: current_term: 1 leader_uuid: "0ebfb31c16f54e308a4bdc944630bd61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebfb31c16f54e308a4bdc944630bd61" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42637 } } peers { permanent_uuid: "8ba7348efe9d4a87bb7e0cfa35140f5a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34037 } } peers { permanent_uuid: "e85eb50b63bd4eb09824856f7241940a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43707 } } }
I20260620 23:35:39.982734 2036 tablet_replica.cc:442] T 231deff76c3f44e683146ae7e15766a5 P 8ba7348efe9d4a87bb7e0cfa35140f5a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:35:39.987615 2037 tablet_replica.cc:439] T 231deff76c3f44e683146ae7e15766a5 P 0ebfb31c16f54e308a4bdc944630bd61: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0ebfb31c16f54e308a4bdc944630bd61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebfb31c16f54e308a4bdc944630bd61" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42637 } } peers { permanent_uuid: "8ba7348efe9d4a87bb7e0cfa35140f5a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34037 } } peers { permanent_uuid: "e85eb50b63bd4eb09824856f7241940a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43707 } } }
I20260620 23:35:39.988085 2037 tablet_replica.cc:442] T 231deff76c3f44e683146ae7e15766a5 P 0ebfb31c16f54e308a4bdc944630bd61: This TxnStatusTablet replica's current role is: LEADER
I20260620 23:35:39.989166 2038 mvcc.cc:205] Tried to move back new op lower bound from 7299066019661156352 to 7299066019529199616. Current Snapshot: MvccSnapshot[applied={T|T < 7299066019661156352}]
I20260620 23:35:39.995169 2039 mvcc.cc:205] Tried to move back new op lower bound from 7299066019661156352 to 7299066019529199616. Current Snapshot: MvccSnapshot[applied={T|T < 7299066019661156352}]
I20260620 23:35:39.996868 2036 tablet_replica.cc:439] T 231deff76c3f44e683146ae7e15766a5 P 8ba7348efe9d4a87bb7e0cfa35140f5a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0ebfb31c16f54e308a4bdc944630bd61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebfb31c16f54e308a4bdc944630bd61" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42637 } } peers { permanent_uuid: "8ba7348efe9d4a87bb7e0cfa35140f5a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34037 } } peers { permanent_uuid: "e85eb50b63bd4eb09824856f7241940a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43707 } } }
I20260620 23:35:39.997125 2036 tablet_replica.cc:442] T 231deff76c3f44e683146ae7e15766a5 P 8ba7348efe9d4a87bb7e0cfa35140f5a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:35:39.998917 2044 tablet_replica.cc:439] T 231deff76c3f44e683146ae7e15766a5 P e85eb50b63bd4eb09824856f7241940a: TxnStatusTablet state changed. Reason: New leader 0ebfb31c16f54e308a4bdc944630bd61. Latest consensus state: current_term: 1 leader_uuid: "0ebfb31c16f54e308a4bdc944630bd61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebfb31c16f54e308a4bdc944630bd61" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42637 } } peers { permanent_uuid: "8ba7348efe9d4a87bb7e0cfa35140f5a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34037 } } peers { permanent_uuid: "e85eb50b63bd4eb09824856f7241940a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43707 } } }
I20260620 23:35:39.999297 2044 tablet_replica.cc:442] T 231deff76c3f44e683146ae7e15766a5 P e85eb50b63bd4eb09824856f7241940a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:35:40.002539 2069 tablet_replica.cc:439] T 231deff76c3f44e683146ae7e15766a5 P 0ebfb31c16f54e308a4bdc944630bd61: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0ebfb31c16f54e308a4bdc944630bd61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebfb31c16f54e308a4bdc944630bd61" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42637 } } peers { permanent_uuid: "8ba7348efe9d4a87bb7e0cfa35140f5a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34037 } } peers { permanent_uuid: "e85eb50b63bd4eb09824856f7241940a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43707 } } }
I20260620 23:35:40.002789 2069 tablet_replica.cc:442] T 231deff76c3f44e683146ae7e15766a5 P 0ebfb31c16f54e308a4bdc944630bd61: This TxnStatusTablet replica's current role is: LEADER
I20260620 23:35:40.007134 2044 tablet_replica.cc:439] T 231deff76c3f44e683146ae7e15766a5 P e85eb50b63bd4eb09824856f7241940a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0ebfb31c16f54e308a4bdc944630bd61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebfb31c16f54e308a4bdc944630bd61" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42637 } } peers { permanent_uuid: "8ba7348efe9d4a87bb7e0cfa35140f5a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34037 } } peers { permanent_uuid: "e85eb50b63bd4eb09824856f7241940a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43707 } } }
I20260620 23:35:40.007434 2044 tablet_replica.cc:442] T 231deff76c3f44e683146ae7e15766a5 P e85eb50b63bd4eb09824856f7241940a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:35:40.038686 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:40.022302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33847 (local address 127.0.97.1:42637)
0620 23:35:40.022532 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:40.022542 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:40.031648 (+ 9106us) server_negotiation.cc:407] Connection header received
0620 23:35:40.031886 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:40.031892 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:40.032003 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:40.032086 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:35:40.032799 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:40.033971 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:40.036483 (+ 2512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:40.037123 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:40.037216 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:40.037857 (+ 641us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:35:40.037996 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:35:40.038055 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:40.038299 (+ 244us) server_negotiation.cc:299] Negotiation successful
0620 23:35:40.038385 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 23:35:40.039646 2078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:40.021781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42637 (local address 127.0.97.3:33847)
0620 23:35:40.031506 (+ 9725us) negotiation.cc:107] Waiting for socket to connect
0620 23:35:40.031525 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:35:40.031732 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:35:40.032248 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:35:40.032254 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:35:40.032266 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:35:40.032591 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:40.032601 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:40.034148 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:40.034153 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:40.036313 (+ 2160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:40.036327 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:40.037382 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:40.037388 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:40.037573 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:40.037646 (+ 73us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:35:40.038141 (+ 495us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:35:40.038150 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 23:35:40.039124 (+ 974us) client_negotiation.cc:240] Negotiation successful
0620 23:35:40.039275 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9523,"thread_start_us":9355,"threads_started":1}
I20260620 23:35:40.056571 1672 raft_consensus.cc:1261] T 17c55b43e18b409d811fd5caede5abfb P 0ebfb31c16f54e308a4bdc944630bd61 [term 1 FOLLOWER]: Refusing update from remote peer e85eb50b63bd4eb09824856f7241940a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:35:40.057296 1815 raft_consensus.cc:1261] T 17c55b43e18b409d811fd5caede5abfb P 8ba7348efe9d4a87bb7e0cfa35140f5a [term 1 FOLLOWER]: Refusing update from remote peer e85eb50b63bd4eb09824856f7241940a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:35:40.057657 2025 consensus_queue.cc:1035] T 17c55b43e18b409d811fd5caede5abfb P e85eb50b63bd4eb09824856f7241940a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfb31c16f54e308a4bdc944630bd61" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:35:40.059064 2025 consensus_queue.cc:1035] T 17c55b43e18b409d811fd5caede5abfb P e85eb50b63bd4eb09824856f7241940a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ba7348efe9d4a87bb7e0cfa35140f5a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:35:40.076200 2035 mvcc.cc:205] Tried to move back new op lower bound from 7299066020061065216 to 7299066019027234816. Current Snapshot: MvccSnapshot[applied={T|T < 7299066020061065216}]
I20260620 23:35:40.126413 2033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:40.112531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33311 (local address 127.0.97.3:43707)
0620 23:35:40.112715 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:40.112721 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:40.119783 (+ 7062us) server_negotiation.cc:407] Connection header received
0620 23:35:40.120015 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:40.120020 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:40.120111 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:40.120220 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:35:40.120780 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:40.121933 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:40.124339 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:40.124912 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:40.125057 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:40.125556 (+ 499us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:35:40.125710 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:35:40.125794 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:40.126007 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 23:35:40.126099 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 23:35:40.127573 2084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:40.112042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43707 (local address 127.0.97.1:33311)
0620 23:35:40.119663 (+ 7621us) negotiation.cc:107] Waiting for socket to connect
0620 23:35:40.119681 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:35:40.119881 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:35:40.120236 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:35:40.120240 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:35:40.120252 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:35:40.120600 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:40.120610 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:40.122108 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:40.122114 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:40.124192 (+ 2078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:40.124210 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:40.125120 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:40.125126 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:40.125300 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:40.125363 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:35:40.125851 (+ 488us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:35:40.125864 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 23:35:40.126967 (+ 1103us) client_negotiation.cc:240] Negotiation successful
0620 23:35:40.127127 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7385,"thread_start_us":7238,"threads_started":1}
I20260620 23:35:40.176004 1515 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:35:40.176292 1515 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:35:40.180487 1515 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b7bda5b0072b4a87b0297077f313def5: Sending DeleteTablet for 3 replicas of tablet 17c55b43e18b409d811fd5caede5abfb
I20260620 23:35:40.181512 1791 tablet_service.cc:1514] Processing DeleteTablet for tablet 17c55b43e18b409d811fd5caede5abfb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:35:40 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53742
I20260620 23:35:40.181672 1652 tablet_service.cc:1514] Processing DeleteTablet for tablet 17c55b43e18b409d811fd5caede5abfb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:35:40 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57654
I20260620 23:35:40.181805 2047 tablet_replica.cc:330] T 17c55b43e18b409d811fd5caede5abfb P 8ba7348efe9d4a87bb7e0cfa35140f5a: stopping tablet replica
I20260620 23:35:40.182018 2047 raft_consensus.cc:2229] T 17c55b43e18b409d811fd5caede5abfb P 8ba7348efe9d4a87bb7e0cfa35140f5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:35:40.182137 1929 tablet_service.cc:1514] Processing DeleteTablet for tablet 17c55b43e18b409d811fd5caede5abfb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:35:40 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46230
I20260620 23:35:40.182236 2047 raft_consensus.cc:2258] T 17c55b43e18b409d811fd5caede5abfb P 8ba7348efe9d4a87bb7e0cfa35140f5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:35:40.182435 2049 tablet_replica.cc:330] T 17c55b43e18b409d811fd5caede5abfb P e85eb50b63bd4eb09824856f7241940a: stopping tablet replica
I20260620 23:35:40.182510 2048 tablet_replica.cc:330] T 17c55b43e18b409d811fd5caede5abfb P 0ebfb31c16f54e308a4bdc944630bd61: stopping tablet replica
I20260620 23:35:40.182647 2049 raft_consensus.cc:2229] T 17c55b43e18b409d811fd5caede5abfb P e85eb50b63bd4eb09824856f7241940a [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:35:40.182705 2048 raft_consensus.cc:2229] T 17c55b43e18b409d811fd5caede5abfb P 0ebfb31c16f54e308a4bdc944630bd61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:35:40.182895 2048 raft_consensus.cc:2258] T 17c55b43e18b409d811fd5caede5abfb P 0ebfb31c16f54e308a4bdc944630bd61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:35:40.183024 2049 raft_consensus.cc:2258] T 17c55b43e18b409d811fd5caede5abfb P e85eb50b63bd4eb09824856f7241940a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:35:40.184547 2048 ts_tablet_manager.cc:1905] T 17c55b43e18b409d811fd5caede5abfb P 0ebfb31c16f54e308a4bdc944630bd61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:35:40.184772 2047 ts_tablet_manager.cc:1905] T 17c55b43e18b409d811fd5caede5abfb P 8ba7348efe9d4a87bb7e0cfa35140f5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:35:40.190869 2049 ts_tablet_manager.cc:1905] T 17c55b43e18b409d811fd5caede5abfb P e85eb50b63bd4eb09824856f7241940a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:35:40.193970 2047 ts_tablet_manager.cc:1918] T 17c55b43e18b409d811fd5caede5abfb P 8ba7348efe9d4a87bb7e0cfa35140f5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:35:40.194161 2047 log.cc:1196] T 17c55b43e18b409d811fd5caede5abfb P 8ba7348efe9d4a87bb7e0cfa35140f5a: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/ts-1/wal/wals/17c55b43e18b409d811fd5caede5abfb
I20260620 23:35:40.194644 2047 ts_tablet_manager.cc:1939] T 17c55b43e18b409d811fd5caede5abfb P 8ba7348efe9d4a87bb7e0cfa35140f5a: Deleting consensus metadata
I20260620 23:35:40.198439 2049 ts_tablet_manager.cc:1918] T 17c55b43e18b409d811fd5caede5abfb P e85eb50b63bd4eb09824856f7241940a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:35:40.200222 1501 catalog_manager.cc:4841] TS 8ba7348efe9d4a87bb7e0cfa35140f5a (127.0.97.2:34037): tablet 17c55b43e18b409d811fd5caede5abfb (table test-table [id=655947f78bfe4c5e910ab5f3c919ba74]) successfully deleted
I20260620 23:35:40.203332 2048 ts_tablet_manager.cc:1918] T 17c55b43e18b409d811fd5caede5abfb P 0ebfb31c16f54e308a4bdc944630bd61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:35:40.203601 2049 log.cc:1196] T 17c55b43e18b409d811fd5caede5abfb P e85eb50b63bd4eb09824856f7241940a: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/ts-2/wal/wals/17c55b43e18b409d811fd5caede5abfb
I20260620 23:35:40.204285 2049 ts_tablet_manager.cc:1939] T 17c55b43e18b409d811fd5caede5abfb P e85eb50b63bd4eb09824856f7241940a: Deleting consensus metadata
I20260620 23:35:40.204530 2048 log.cc:1196] T 17c55b43e18b409d811fd5caede5abfb P 0ebfb31c16f54e308a4bdc944630bd61: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998525501163-388-0/minicluster-data/ts-0/wal/wals/17c55b43e18b409d811fd5caede5abfb
I20260620 23:35:40.205948 1502 catalog_manager.cc:4841] TS e85eb50b63bd4eb09824856f7241940a (127.0.97.3:43707): tablet 17c55b43e18b409d811fd5caede5abfb (table test-table [id=655947f78bfe4c5e910ab5f3c919ba74]) successfully deleted
I20260620 23:35:40.206663 2048 ts_tablet_manager.cc:1939] T 17c55b43e18b409d811fd5caede5abfb P 0ebfb31c16f54e308a4bdc944630bd61: Deleting consensus metadata
I20260620 23:35:40.207930 1500 catalog_manager.cc:4841] TS 0ebfb31c16f54e308a4bdc944630bd61 (127.0.97.1:42637): tablet 17c55b43e18b409d811fd5caede5abfb (table test-table [id=655947f78bfe4c5e910ab5f3c919ba74]) successfully deleted
Jun 20 23:35:40 dist-test-slave-tcgx krb5kdc[1470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998539, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 23:35:40.234923 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:40.196547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57670 (local address 127.0.97.1:42637)
0620 23:35:40.196768 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:40.196774 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:40.213351 (+ 16577us) server_negotiation.cc:407] Connection header received
0620 23:35:40.213639 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:40.213644 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:40.213732 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:40.213820 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:35:40.215834 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:40.217044 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:40.219084 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:40.219898 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:40.220043 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:40.225922 (+ 5879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:35:40.225943 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:35:40.225964 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:35:40.226019 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:35:40.231274 (+ 5255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:40.232293 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:40.232304 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:40.232308 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:40.232505 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:40.232985 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:40.232990 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:40.232994 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:40.233559 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:35:40.233914 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:40.234243 (+ 329us) server_negotiation.cc:299] Negotiation successful
0620 23:35:40.234540 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260620 23:35:40.235821 1717 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57670
I20260620 23:35:40.264839 2062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:40.244197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60404 (local address 127.0.97.62:44817)
0620 23:35:40.244452 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:40.244459 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:40.247927 (+ 3468us) server_negotiation.cc:407] Connection header received
0620 23:35:40.248155 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:40.248163 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:40.248249 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:40.248360 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:35:40.250188 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:40.251367 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:40.253258 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:40.253915 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:40.254045 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:40.257285 (+ 3240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:35:40.257294 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:35:40.257301 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:35:40.257347 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:35:40.261134 (+ 3787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:40.262020 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:40.262027 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:40.262031 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:40.262645 (+ 614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:40.263142 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:40.263148 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:40.263151 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:40.263569 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:35:40.263834 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:40.264226 (+ 392us) server_negotiation.cc:299] Negotiation successful
0620 23:35:40.264483 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
W20260620 23:35:40.265586 1515 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60404
I20260620 23:35:40.267872 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 1586
I20260620 23:35:40.287448 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:40.269162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57678 (local address 127.0.97.1:42637)
0620 23:35:40.269374 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:40.269382 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:40.270431 (+ 1049us) server_negotiation.cc:407] Connection header received
0620 23:35:40.270566 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:40.270573 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:40.270663 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:40.271216 (+ 553us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:35:40.272610 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:40.273671 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:40.275434 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:40.276088 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:40.276224 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:40.279455 (+ 3231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:35:40.279463 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:35:40.279468 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:35:40.279513 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:35:40.283870 (+ 4357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:40.284937 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:40.284945 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:40.284949 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:40.285124 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:40.285598 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:40.285604 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:40.285607 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:40.285997 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:35:40.286241 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:40.286816 (+ 575us) server_negotiation.cc:299] Negotiation successful
0620 23:35:40.287101 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260620 23:35:40.289067 1717 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57678
W20260620 23:35:40.289714 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 1586: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:35:40.289921 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 1586
I20260620 23:35:40.315572 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 1726
Jun 20 23:35:40 dist-test-slave-tcgx krb5kdc[1470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998539, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 23:35:40.331892 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:40.316447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53748 (local address 127.0.97.2:34037)
0620 23:35:40.316650 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:40.316656 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:40.316677 (+ 21us) server_negotiation.cc:407] Connection header received
0620 23:35:40.316820 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:40.316827 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:40.316896 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:40.317012 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:35:40.318183 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:40.319021 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:40.320324 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:40.320757 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:40.320898 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:40.325670 (+ 4772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:35:40.325685 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:35:40.325701 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:35:40.325739 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:35:40.328890 (+ 3151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:40.329662 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:40.329670 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:40.329672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:40.329813 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:40.330205 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:40.330210 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:40.330212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:40.330552 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:35:40.330885 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:40.331183 (+ 298us) server_negotiation.cc:299] Negotiation successful
0620 23:35:40.331406 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260620 23:35:40.332680 1856 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53748
W20260620 23:35:40.333441 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 1726: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:35:40.333614 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 1726
I20260620 23:35:40.358510 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 1865
Jun 20 23:35:40 dist-test-slave-tcgx krb5kdc[1470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998539, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 23:35:40.375981 2033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:40.359384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46252 (local address 127.0.97.3:43707)
0620 23:35:40.359565 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:40.359570 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:40.359624 (+ 54us) server_negotiation.cc:407] Connection header received
0620 23:35:40.359842 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:40.359847 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:40.359912 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:40.360024 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:35:40.361642 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:40.362340 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:40.364164 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:40.364628 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:40.364737 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:40.369982 (+ 5245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:35:40.369999 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:35:40.370015 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:35:40.370054 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:35:40.373106 (+ 3052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:40.373773 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:40.373780 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:40.373783 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:40.373915 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:40.374242 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:40.374245 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:40.374247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:40.374620 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:35:40.374848 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:40.375410 (+ 562us) server_negotiation.cc:299] Negotiation successful
0620 23:35:40.375594 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260620 23:35:40.376786 1995 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46252
W20260620 23:35:40.377620 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 1865: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:35:40.377826 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 1865
I20260620 23:35:40.398813 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 1483
I20260620 23:35:40.400103 1555 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:35:40.513689 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 1483
2026-06-20T23:35:40Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4953 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:35:40 dist-test-slave-tcgx krb5kdc[2094](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:35:40 dist-test-slave-tcgx krb5kdc[2094](info): set up 2 sockets
Jun 20 23:35:40 dist-test-slave-tcgx krb5kdc[2094](info): commencing operation
krb5kdc: starting...
W20260620 23:35:41.603062 388 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.005s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/krb5kdc/test-user.keytab.
Jun 20 23:35:41 dist-test-slave-tcgx krb5kdc[2094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:35:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:35:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 23:35:41.790877 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43025
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43025
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:35:42.110795 2107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:35:42.111245 2107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:35:42.111413 2107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:35:42.118811 2107 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:35:42.119007 2107 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 23:35:42.119107 2107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:35:42.119220 2107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:35:42.119313 2107 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:35:42.119400 2107 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:35:42.129727 2107 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43025
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:35:42.131178 2107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:35:42.132622 2107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:35:42.144667 2112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:35:42.144732 2113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:35:42.144707 2115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:35:42.145431 2107 server_base.cc:1034] running on GCE node
I20260620 23:35:42.146194 2107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:35:42.147797 2107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:35:42.149008 2107 hybrid_clock.cc:648] HybridClock initialized: now 1781998542148924 us; error 65 us; skew 500 ppm
I20260620 23:35:42.149304 2107 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:35:42 dist-test-slave-tcgx krb5kdc[2094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998542, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:35:42.152428 2107 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 23:35:42.153939 2107 webserver.cc:460] Webserver started at http://127.0.97.62:32863/ using document root <none> and password file <none>
I20260620 23:35:42.154613 2107 fs_manager.cc:359] Metadata directory not provided
I20260620 23:35:42.154703 2107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:35:42.155035 2107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:35:42.158023 2107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/master-0/data/instance:
uuid: "78b3f3598f0540858739235691fbe934"
format_stamp: "Formatted at 2026-06-20 23:35:42 on dist-test-slave-tcgx"
server_key: "5c0b4d0b3a2ae4d3b327ec196a09c445"
server_key_iv: "8260a484f684fbb3ec75326d37aa1b16"
server_key_version: "encryptionkey@0"
I20260620 23:35:42.158730 2107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/master-0/wal/instance:
uuid: "78b3f3598f0540858739235691fbe934"
format_stamp: "Formatted at 2026-06-20 23:35:42 on dist-test-slave-tcgx"
server_key: "5c0b4d0b3a2ae4d3b327ec196a09c445"
server_key_iv: "8260a484f684fbb3ec75326d37aa1b16"
server_key_version: "encryptionkey@0"
I20260620 23:35:42.163995 2107 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 23:35:42.167502 2122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:35:42.168946 2107 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 23:35:42.169086 2107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/master-0/wal
uuid: "78b3f3598f0540858739235691fbe934"
format_stamp: "Formatted at 2026-06-20 23:35:42 on dist-test-slave-tcgx"
server_key: "5c0b4d0b3a2ae4d3b327ec196a09c445"
server_key_iv: "8260a484f684fbb3ec75326d37aa1b16"
server_key_version: "encryptionkey@0"
I20260620 23:35:42.169263 2107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:35:42.180090 2107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:35:42.184595 2107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:35:42.184828 2107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:35:42.206673 2107 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43025
I20260620 23:35:42.206681 2183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43025 every 8 connection(s)
I20260620 23:35:42.208102 2107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:35:42.211918 2184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:35:42.218453 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 2107
I20260620 23:35:42.218706 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/master-0/wal/instance
I20260620 23:35:42.219571 388 external_mini_cluster.cc:1428] Setting key 762167211000cef9990dc6334023ee6f
I20260620 23:35:42.223451 2184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78b3f3598f0540858739235691fbe934: Bootstrap starting.
I20260620 23:35:42.226791 2184 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78b3f3598f0540858739235691fbe934: Neither blocks nor log segments found. Creating new log.
I20260620 23:35:42.227989 2184 log.cc:824] T 00000000000000000000000000000000 P 78b3f3598f0540858739235691fbe934: Log is configured to *not* fsync() on all Append() calls
I20260620 23:35:42.231232 2184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78b3f3598f0540858739235691fbe934: No bootstrap required, opened a new log
Jun 20 23:35:42 dist-test-slave-tcgx krb5kdc[2094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:35:42.243489 2184 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78b3f3598f0540858739235691fbe934 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78b3f3598f0540858739235691fbe934" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43025 } }
I20260620 23:35:42.244011 2184 raft_consensus.cc:374] T 00000000000000000000000000000000 P 78b3f3598f0540858739235691fbe934 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:35:42.244132 2184 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78b3f3598f0540858739235691fbe934 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b3f3598f0540858739235691fbe934, State: Initialized, Role: FOLLOWER
I20260620 23:35:42.244879 2184 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78b3f3598f0540858739235691fbe934 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78b3f3598f0540858739235691fbe934" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43025 } }
I20260620 23:35:42.245190 2184 raft_consensus.cc:388] T 00000000000000000000000000000000 P 78b3f3598f0540858739235691fbe934 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:35:42.245342 2184 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78b3f3598f0540858739235691fbe934 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:35:42.245486 2187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:42.221730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37174 (local address 127.0.97.62:43025)
0620 23:35:42.222545 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:42.222558 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:42.222600 (+ 42us) server_negotiation.cc:407] Connection header received
0620 23:35:42.224503 (+ 1903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:42.224530 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:42.224986 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:42.225423 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:35:42.226906 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:42.228413 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:42.229793 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:42.230535 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:42.230745 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:42.235823 (+ 5078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:35:42.235838 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:35:42.235869 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:35:42.235929 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:35:42.240435 (+ 4506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:42.241179 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:42.241189 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:42.241201 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:42.241454 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:42.241845 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:42.241855 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:42.241859 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:42.242677 (+ 818us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:35:42.243032 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:42.243438 (+ 406us) server_negotiation.cc:299] Negotiation successful
0620 23:35:42.243924 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":377,"threads_started":1}
I20260620 23:35:42.245563 2184 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78b3f3598f0540858739235691fbe934 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:35:42.248317 2184 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78b3f3598f0540858739235691fbe934 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78b3f3598f0540858739235691fbe934" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43025 } }
I20260620 23:35:42.248804 2184 leader_election.cc:302] T 00000000000000000000000000000000 P 78b3f3598f0540858739235691fbe934 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 78b3f3598f0540858739235691fbe934; no voters:
I20260620 23:35:42.249320 2184 leader_election.cc:288] T 00000000000000000000000000000000 P 78b3f3598f0540858739235691fbe934 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:35:42.249477 2189 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78b3f3598f0540858739235691fbe934 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:35:42.250200 2189 raft_consensus.cc:686] T 00000000000000000000000000000000 P 78b3f3598f0540858739235691fbe934 [term 1 LEADER]: Becoming Leader. State: Replica: 78b3f3598f0540858739235691fbe934, State: Running, Role: LEADER
I20260620 23:35:42.250746 2189 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78b3f3598f0540858739235691fbe934 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78b3f3598f0540858739235691fbe934" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43025 } }
I20260620 23:35:42.251549 2184 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78b3f3598f0540858739235691fbe934 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:35:42.254710 2191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78b3f3598f0540858739235691fbe934 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78b3f3598f0540858739235691fbe934. Latest consensus state: current_term: 1 leader_uuid: "78b3f3598f0540858739235691fbe934" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78b3f3598f0540858739235691fbe934" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43025 } } }
I20260620 23:35:42.255332 2191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78b3f3598f0540858739235691fbe934 [sys.catalog]: This master's current role is: LEADER
I20260620 23:35:42.254678 2190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78b3f3598f0540858739235691fbe934 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "78b3f3598f0540858739235691fbe934" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78b3f3598f0540858739235691fbe934" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43025 } } }
I20260620 23:35:42.256805 2190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78b3f3598f0540858739235691fbe934 [sys.catalog]: This master's current role is: LEADER
I20260620 23:35:42.258479 2197 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:35:42.264706 2197 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:35:42.274331 2197 catalog_manager.cc:1347] Generated new cluster ID: 941c69a0aa604beebef983e80528881f
I20260620 23:35:42.274469 2197 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:35:42.291590 2197 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:35:42.292972 2197 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:35:42.303874 2197 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 78b3f3598f0540858739235691fbe934: Generated new TSK 0
I20260620 23:35:42.304834 2197 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 23:35:42.354578 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43025
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:35:42.691027 2210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:35:42.691360 2210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:35:42.691521 2210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:35:42.698885 2210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:35:42.699127 2210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:35:42.699290 2210 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 23:35:42.699359 2210 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 23:35:42.699481 2210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 23:35:42.709676 2210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:43025
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:35:42.711133 2210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:35:42.712539 2210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:35:42.725576 2215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:35:42.725546 2216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:35:42.725459 2218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:35:42.726576 2210 server_base.cc:1034] running on GCE node
I20260620 23:35:42.727283 2210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:35:42.728262 2210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:35:42.729494 2210 hybrid_clock.cc:648] HybridClock initialized: now 1781998542729432 us; error 61 us; skew 500 ppm
I20260620 23:35:42.729820 2210 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:35:42 dist-test-slave-tcgx krb5kdc[2094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998542, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:35:42.733072 2210 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 23:35:42.734616 2210 webserver.cc:460] Webserver started at http://127.0.97.1:33325/ using document root <none> and password file <none>
I20260620 23:35:42.735378 2210 fs_manager.cc:359] Metadata directory not provided
I20260620 23:35:42.735483 2210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:35:42.735738 2210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:35:42.738703 2210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/ts-0/data/instance:
uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5"
format_stamp: "Formatted at 2026-06-20 23:35:42 on dist-test-slave-tcgx"
server_key: "cd691358ad23c4bd1c86efff67e701cc"
server_key_iv: "ea346760d56efc8c4083171a9e4221e8"
server_key_version: "encryptionkey@0"
I20260620 23:35:42.739532 2210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/ts-0/wal/instance:
uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5"
format_stamp: "Formatted at 2026-06-20 23:35:42 on dist-test-slave-tcgx"
server_key: "cd691358ad23c4bd1c86efff67e701cc"
server_key_iv: "ea346760d56efc8c4083171a9e4221e8"
server_key_version: "encryptionkey@0"
I20260620 23:35:42.744947 2210 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260620 23:35:42.748961 2225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:35:42.750469 2210 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 23:35:42.750677 2210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/ts-0/wal
uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5"
format_stamp: "Formatted at 2026-06-20 23:35:42 on dist-test-slave-tcgx"
server_key: "cd691358ad23c4bd1c86efff67e701cc"
server_key_iv: "ea346760d56efc8c4083171a9e4221e8"
server_key_version: "encryptionkey@0"
I20260620 23:35:42.750900 2210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:35:42.780833 2210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:35:42.785283 2210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:35:42.785593 2210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:35:42.788360 2210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:35:42.788512 2210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:35:42.788662 2210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:35:42.788772 2210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 23:35:42 dist-test-slave-tcgx krb5kdc[2094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998542, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:35:42.814309 2234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:42.788937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53505 (local address 127.0.97.62:43025)
0620 23:35:42.789670 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:42.789678 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:42.790827 (+ 1149us) server_negotiation.cc:407] Connection header received
0620 23:35:42.793380 (+ 2553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:42.793386 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:42.793478 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:42.793603 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:35:42.795979 (+ 2376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:42.796914 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:42.799378 (+ 2464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:42.799964 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:42.800115 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:42.805685 (+ 5570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:35:42.805694 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:35:42.805699 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:35:42.805762 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:35:42.809049 (+ 3287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:42.810454 (+ 1405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:42.810462 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:42.810467 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:42.810621 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:42.811285 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:42.811293 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:42.811299 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:42.811722 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:35:42.811891 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:42.813722 (+ 1831us) server_negotiation.cc:299] Negotiation successful
0620 23:35:42.813929 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":476,"threads_started":1}
I20260620 23:35:42.815452 2235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:42.789682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43025 (local address 127.0.97.1:53505)
0620 23:35:42.790621 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0620 23:35:42.790687 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0620 23:35:42.793050 (+ 2363us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:35:42.793900 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:35:42.793917 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:35:42.794771 (+ 854us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:35:42.795778 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:42.795812 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:42.797108 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:42.797112 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:42.799215 (+ 2103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:42.799228 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:42.800241 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:42.800249 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:42.800479 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:42.801439 (+ 960us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:35:42.801477 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:35:42.805307 (+ 3830us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:35:42.809298 (+ 3991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:35:42.809313 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:35:42.809339 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:35:42.810307 (+ 968us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:35:42.810762 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:35:42.810768 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:35:42.810772 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:35:42.811114 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:35:42.812000 (+ 886us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:35:42.812013 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:35:42.812652 (+ 639us) client_negotiation.cc:769] Sending connection context
0620 23:35:42.813063 (+ 411us) client_negotiation.cc:240] Negotiation successful
0620 23:35:42.813813 (+ 750us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":404,"threads_started":1}
I20260620 23:35:42.834179 2210 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40819
I20260620 23:35:42.834231 2345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40819 every 8 connection(s)
I20260620 23:35:42.835759 2210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
I20260620 23:35:42.838300 2346 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43025
I20260620 23:35:42.838541 2346 heartbeater.cc:464] Registering TS with master...
I20260620 23:35:42.839277 2346 heartbeater.cc:511] Master 127.0.97.62:43025 requested a full tablet report, sending...
I20260620 23:35:42.841706 2139 ts_manager.cc:194] Registered new tserver with Master: 1fcac2741ad84a55bc68e5dd99ae3bd5 (127.0.97.1:40819)
I20260620 23:35:42.842386 2327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:42.826809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43025 (local address 127.0.97.1:40051)
0620 23:35:42.827755 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0620 23:35:42.827771 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:35:42.827935 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:35:42.828386 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:35:42.828394 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:35:42.829255 (+ 861us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:35:42.829864 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:42.829882 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:42.831565 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:42.831572 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:42.833081 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:42.833098 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:42.834201 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:42.834214 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:42.834411 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:42.835199 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:35:42.835216 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:35:42.836066 (+ 850us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:35:42.839943 (+ 3877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:35:42.839951 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:35:42.839956 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:35:42.840341 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:35:42.840776 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:35:42.840779 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:35:42.840781 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:35:42.840886 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:35:42.841533 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:35:42.841536 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:35:42.841676 (+ 140us) client_negotiation.cc:769] Sending connection context
0620 23:35:42.841783 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 23:35:42.842085 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":806,"thread_start_us":553,"threads_started":1}
I20260620 23:35:42.842708 2234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:42.826635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40051 (local address 127.0.97.62:43025)
0620 23:35:42.826810 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:42.826814 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:42.827869 (+ 1055us) server_negotiation.cc:407] Connection header received
0620 23:35:42.828093 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:42.828101 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:42.828201 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:42.828302 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:35:42.830090 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:42.831391 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:42.833261 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:42.833967 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:42.834107 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:42.836271 (+ 2164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:35:42.836281 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:35:42.836289 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:35:42.836340 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:35:42.839809 (+ 3469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:42.840462 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:42.840472 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:42.840477 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:42.840657 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:42.840995 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:42.841000 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:42.841003 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:42.841396 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:35:42.841572 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:42.841937 (+ 365us) server_negotiation.cc:299] Negotiation successful
0620 23:35:42.842174 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 23:35:42.843743 2139 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:53505
I20260620 23:35:42.844409 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 2210
I20260620 23:35:42.844630 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/ts-0/wal/instance
I20260620 23:35:42.845376 388 external_mini_cluster.cc:1428] Setting key e74339728709ee9736acc5d54dcd2be6
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 23:35:42.884611 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43025
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:35:43.214535 2350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:35:43.214838 2350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:35:43.215066 2350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:35:43.223141 2350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:35:43.223258 2350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:35:43.223407 2350 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 23:35:43.223486 2350 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 23:35:43.223603 2350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 23:35:43.233857 2350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:43025
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:35:43.235445 2350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:35:43.236862 2350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:35:43.250509 2355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:35:43.250520 2356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:35:43.250998 2358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:35:43.251446 2350 server_base.cc:1034] running on GCE node
I20260620 23:35:43.252182 2350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:35:43.253170 2350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:35:43.254415 2350 hybrid_clock.cc:648] HybridClock initialized: now 1781998543254362 us; error 42 us; skew 500 ppm
I20260620 23:35:43.254822 2350 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:35:43 dist-test-slave-tcgx krb5kdc[2094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998543, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:35:43.258203 2350 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 23:35:43.259999 2350 webserver.cc:460] Webserver started at http://127.0.97.2:40393/ using document root <none> and password file <none>
I20260620 23:35:43.260685 2350 fs_manager.cc:359] Metadata directory not provided
I20260620 23:35:43.260785 2350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:35:43.261117 2350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:35:43.264137 2350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/ts-1/data/instance:
uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6"
format_stamp: "Formatted at 2026-06-20 23:35:43 on dist-test-slave-tcgx"
server_key: "8d608b353b92c44ab62645784be15786"
server_key_iv: "187be7d9438d5df5f79ec106011acdda"
server_key_version: "encryptionkey@0"
I20260620 23:35:43.264906 2350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/ts-1/wal/instance:
uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6"
format_stamp: "Formatted at 2026-06-20 23:35:43 on dist-test-slave-tcgx"
server_key: "8d608b353b92c44ab62645784be15786"
server_key_iv: "187be7d9438d5df5f79ec106011acdda"
server_key_version: "encryptionkey@0"
I20260620 23:35:43.270051 2350 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 23:35:43.273825 2365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:35:43.275302 2350 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:35:43.275442 2350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/ts-1/wal
uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6"
format_stamp: "Formatted at 2026-06-20 23:35:43 on dist-test-slave-tcgx"
server_key: "8d608b353b92c44ab62645784be15786"
server_key_iv: "187be7d9438d5df5f79ec106011acdda"
server_key_version: "encryptionkey@0"
I20260620 23:35:43.275614 2350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:35:43.292172 2350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:35:43.296334 2350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:35:43.296577 2350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:35:43.299340 2350 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:35:43.299466 2350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:35:43.299599 2350 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:35:43.299700 2350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 23:35:43 dist-test-slave-tcgx krb5kdc[2094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998543, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:35:43.323412 2234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:43.299845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43037 (local address 127.0.97.62:43025)
0620 23:35:43.300036 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:43.300045 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:43.301746 (+ 1701us) server_negotiation.cc:407] Connection header received
0620 23:35:43.303911 (+ 2165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:43.303918 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:43.304031 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:43.304146 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:35:43.306497 (+ 2351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:43.307876 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:43.310670 (+ 2794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:43.311486 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:43.311670 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:43.316864 (+ 5194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:35:43.316870 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:35:43.316874 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:35:43.316910 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:35:43.320170 (+ 3260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:43.320991 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:43.320996 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:43.321000 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:43.321129 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:43.321590 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:43.321597 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:43.321600 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:43.321897 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:35:43.322046 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:43.322793 (+ 747us) server_negotiation.cc:299] Negotiation successful
0620 23:35:43.323054 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 23:35:43.325385 2374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:43.300594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43025 (local address 127.0.97.2:43037)
0620 23:35:43.301477 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0620 23:35:43.301564 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0620 23:35:43.303604 (+ 2040us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:35:43.304432 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:35:43.304449 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:35:43.305322 (+ 873us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:35:43.306245 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:43.306264 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:43.308085 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:43.308093 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:43.310495 (+ 2402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:43.310513 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:43.311852 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:43.311861 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:43.312113 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:43.313045 (+ 932us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:35:43.313074 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:35:43.316563 (+ 3489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:35:43.320344 (+ 3781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:35:43.320355 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:35:43.320384 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:35:43.320851 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:35:43.321249 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:35:43.321254 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:35:43.321257 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:35:43.321469 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:35:43.322036 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:35:43.322045 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:35:43.322432 (+ 387us) client_negotiation.cc:769] Sending connection context
0620 23:35:43.322719 (+ 287us) client_negotiation.cc:240] Negotiation successful
0620 23:35:43.323352 (+ 633us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":381,"threads_started":1}
I20260620 23:35:43.344338 2350 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33641
I20260620 23:35:43.344430 2484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33641 every 8 connection(s)
I20260620 23:35:43.345861 2350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
I20260620 23:35:43.348063 2485 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43025
I20260620 23:35:43.348291 2485 heartbeater.cc:464] Registering TS with master...
I20260620 23:35:43.349139 2485 heartbeater.cc:511] Master 127.0.97.62:43025 requested a full tablet report, sending...
I20260620 23:35:43.350318 2459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:43.334163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43025 (local address 127.0.97.2:40385)
0620 23:35:43.335281 (+ 1118us) negotiation.cc:107] Waiting for socket to connect
0620 23:35:43.335301 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:35:43.335527 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:35:43.335910 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:35:43.335914 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:35:43.336605 (+ 691us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:35:43.337145 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:43.337167 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:43.338579 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:43.338585 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:43.340429 (+ 1844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:43.340440 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:43.341516 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:43.341526 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:43.341754 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:43.342573 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:35:43.342594 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:35:43.343888 (+ 1294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:35:43.347610 (+ 3722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:35:43.347614 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:35:43.347618 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:35:43.348121 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:35:43.348548 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:35:43.348551 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:35:43.348554 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:35:43.348675 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:35:43.349369 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:35:43.349373 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:35:43.349524 (+ 151us) client_negotiation.cc:769] Sending connection context
0620 23:35:43.349628 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 23:35:43.349862 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":887,"thread_start_us":621,"threads_started":1}
I20260620 23:35:43.350344 2234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:43.333971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40385 (local address 127.0.97.62:43025)
0620 23:35:43.334185 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:43.334192 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:43.335411 (+ 1219us) server_negotiation.cc:407] Connection header received
0620 23:35:43.335691 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:43.335698 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:43.335779 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:43.335885 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:35:43.337303 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:43.338454 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:43.340597 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:43.341269 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:43.341398 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:43.344161 (+ 2763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:35:43.344170 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:35:43.344177 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:35:43.344225 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:35:43.347453 (+ 3228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:43.348288 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:43.348293 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:43.348296 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:43.348438 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:43.348803 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:43.348811 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:43.348815 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:43.349256 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:35:43.349396 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:43.349773 (+ 377us) server_negotiation.cc:299] Negotiation successful
0620 23:35:43.350003 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 23:35:43.350688 2139 ts_manager.cc:194] Registered new tserver with Master: f10804aa6b1d4b22a5a82fb8e2e9d0d6 (127.0.97.2:33641)
I20260620 23:35:43.351658 2139 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:43037
I20260620 23:35:43.353336 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 2350
I20260620 23:35:43.353524 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/ts-1/wal/instance
I20260620 23:35:43.354061 388 external_mini_cluster.cc:1428] Setting key a74aa11f11b8ee609c0c6f5261cb7dac
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 23:35:43.396907 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43025
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:35:43.728996 2489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:35:43.729295 2489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:35:43.729398 2489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:35:43.736838 2489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:35:43.736966 2489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:35:43.737052 2489 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 23:35:43.737104 2489 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 23:35:43.737172 2489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 23:35:43.747385 2489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:43025
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:35:43.748667 2489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:35:43.749934 2489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:35:43.763541 2495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:35:43.763700 2497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:35:43.764050 2494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:35:43.764524 2489 server_base.cc:1034] running on GCE node
I20260620 23:35:43.765196 2489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:35:43.766225 2489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:35:43.767457 2489 hybrid_clock.cc:648] HybridClock initialized: now 1781998543767395 us; error 53 us; skew 500 ppm
I20260620 23:35:43.767786 2489 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:35:43 dist-test-slave-tcgx krb5kdc[2094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998543, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:35:43.770843 2489 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 23:35:43.772468 2489 webserver.cc:460] Webserver started at http://127.0.97.3:35917/ using document root <none> and password file <none>
I20260620 23:35:43.773344 2489 fs_manager.cc:359] Metadata directory not provided
I20260620 23:35:43.773496 2489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:35:43.773813 2489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:35:43.776803 2489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/ts-2/data/instance:
uuid: "fa4063ffe74c4ef183d404cf1d2c20cd"
format_stamp: "Formatted at 2026-06-20 23:35:43 on dist-test-slave-tcgx"
server_key: "c27e1fc273f14380b81d52d9d19dedd8"
server_key_iv: "deedc841a8c55994e10944b36a79bd2d"
server_key_version: "encryptionkey@0"
I20260620 23:35:43.777602 2489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/ts-2/wal/instance:
uuid: "fa4063ffe74c4ef183d404cf1d2c20cd"
format_stamp: "Formatted at 2026-06-20 23:35:43 on dist-test-slave-tcgx"
server_key: "c27e1fc273f14380b81d52d9d19dedd8"
server_key_iv: "deedc841a8c55994e10944b36a79bd2d"
server_key_version: "encryptionkey@0"
I20260620 23:35:43.783098 2489 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 23:35:43.786711 2504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:35:43.788223 2489 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:35:43.788403 2489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/ts-2/wal
uuid: "fa4063ffe74c4ef183d404cf1d2c20cd"
format_stamp: "Formatted at 2026-06-20 23:35:43 on dist-test-slave-tcgx"
server_key: "c27e1fc273f14380b81d52d9d19dedd8"
server_key_iv: "deedc841a8c55994e10944b36a79bd2d"
server_key_version: "encryptionkey@0"
I20260620 23:35:43.788615 2489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:35:43.837365 2489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:35:43.843937 2489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:35:43.844338 2489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:35:43.847793 2346 heartbeater.cc:503] Master 127.0.97.62:43025 was elected leader, sending a full tablet report...
I20260620 23:35:43.848403 2489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:35:43.848541 2489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:35:43.848651 2489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:35:43.848717 2489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 23:35:43 dist-test-slave-tcgx krb5kdc[2094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998543, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:35:43.877180 2513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:43.849835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43025 (local address 127.0.97.3:48963)
0620 23:35:43.850790 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0620 23:35:43.850849 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 23:35:43.853291 (+ 2442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:35:43.854030 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:35:43.854039 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:35:43.854798 (+ 759us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:35:43.855590 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:43.855625 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:43.857093 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:43.857097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:43.858463 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:43.858474 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:43.859756 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:43.859761 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:43.860038 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:43.864087 (+ 4049us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:35:43.864130 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:35:43.867511 (+ 3381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:35:43.871647 (+ 4136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:35:43.871664 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:35:43.871695 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:35:43.872561 (+ 866us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:35:43.873172 (+ 611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:35:43.873180 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:35:43.873186 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:35:43.873542 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:35:43.874052 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:35:43.874058 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:35:43.874648 (+ 590us) client_negotiation.cc:769] Sending connection context
0620 23:35:43.874981 (+ 333us) client_negotiation.cc:240] Negotiation successful
0620 23:35:43.875531 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":399,"threads_started":1}
I20260620 23:35:43.878057 2234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:43.848745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48963 (local address 127.0.97.62:43025)
0620 23:35:43.848928 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:43.848935 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:43.850981 (+ 2046us) server_negotiation.cc:407] Connection header received
0620 23:35:43.853561 (+ 2580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:43.853567 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:43.853665 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:43.853783 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:35:43.855883 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:43.856933 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:43.858607 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:43.859252 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:43.859421 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:43.867873 (+ 8452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:35:43.867882 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:35:43.867904 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:35:43.867959 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:35:43.871459 (+ 3500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:43.872781 (+ 1322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:43.872790 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:43.872797 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:43.873001 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:43.873668 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:43.873680 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:43.873689 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:43.873957 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:35:43.874073 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:43.877359 (+ 3286us) server_negotiation.cc:299] Negotiation successful
0620 23:35:43.877658 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 23:35:43.895743 2489 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37077
I20260620 23:35:43.895743 2623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37077 every 8 connection(s)
I20260620 23:35:43.897157 2489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
I20260620 23:35:43.899317 2624 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43025
I20260620 23:35:43.899544 2624 heartbeater.cc:464] Registering TS with master...
I20260620 23:35:43.900426 2624 heartbeater.cc:511] Master 127.0.97.62:43025 requested a full tablet report, sending...
I20260620 23:35:43.901953 2139 ts_manager.cc:194] Registered new tserver with Master: fa4063ffe74c4ef183d404cf1d2c20cd (127.0.97.3:37077)
I20260620 23:35:43.902355 2596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:43.885682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43025 (local address 127.0.97.3:35445)
0620 23:35:43.886513 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0620 23:35:43.886539 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 23:35:43.886743 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:35:43.887269 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:35:43.887276 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:35:43.887909 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:35:43.888450 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:43.888469 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:43.889764 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:43.889771 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:43.892306 (+ 2535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:43.892326 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:43.893780 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:43.893787 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:43.893941 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:43.894715 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:35:43.894741 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:35:43.896205 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:35:43.899790 (+ 3585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:35:43.899795 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:35:43.899799 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:35:43.900351 (+ 552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:35:43.900735 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:35:43.900740 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:35:43.900745 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:35:43.900890 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:35:43.901428 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:35:43.901433 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:35:43.901621 (+ 188us) client_negotiation.cc:769] Sending connection context
0620 23:35:43.901753 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 23:35:43.901973 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":434,"threads_started":1}
I20260620 23:35:43.902613 2234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:43.885606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35445 (local address 127.0.97.62:43025)
0620 23:35:43.885849 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:43.885855 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:43.886653 (+ 798us) server_negotiation.cc:407] Connection header received
0620 23:35:43.887014 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:43.887021 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:43.887113 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:43.887203 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:35:43.888641 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:43.889571 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:43.892540 (+ 2969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:43.893346 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:43.893486 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:43.896386 (+ 2900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:35:43.896392 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:35:43.896395 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:35:43.896427 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:35:43.899666 (+ 3239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:43.900459 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:43.900463 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:43.900465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:43.900624 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:43.900993 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:43.900996 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:43.900998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:43.901308 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:35:43.901449 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:43.901900 (+ 451us) server_negotiation.cc:299] Negotiation successful
0620 23:35:43.902129 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 23:35:43.903052 2139 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:48963
I20260620 23:35:43.906023 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 2489
I20260620 23:35:43.906224 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/ts-2/wal/instance
I20260620 23:35:43.906653 388 external_mini_cluster.cc:1428] Setting key e85435e859db69aa923778f3fbb7c7f2
I20260620 23:35:43.909549 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 23:35:43 dist-test-slave-tcgx krb5kdc[2094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 23:35:43 dist-test-slave-tcgx krb5kdc[2094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:35:43.959056 2234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:43.943942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37178 (local address 127.0.97.62:43025)
0620 23:35:43.944231 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:43.944237 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:43.944453 (+ 216us) server_negotiation.cc:407] Connection header received
0620 23:35:43.944681 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:43.944691 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:43.944762 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:43.944874 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:35:43.946308 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:43.947133 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:43.948374 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:43.948774 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:43.948906 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:43.953223 (+ 4317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:35:43.953235 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:35:43.953239 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:35:43.953276 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:35:43.956346 (+ 3070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:43.957037 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:43.957043 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:43.957051 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:43.957189 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:43.957575 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:43.957582 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:43.957585 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:43.957854 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:35:43.958119 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:43.958429 (+ 310us) server_negotiation.cc:299] Negotiation successful
0620 23:35:43.958607 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20260620 23:35:43.963200 2138 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37178:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:35:43.970129 2138 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:35:43.994429 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:43.983376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33641 (local address 127.0.0.1:37478)
0620 23:35:43.985943 (+ 2567us) negotiation.cc:107] Waiting for socket to connect
0620 23:35:43.985962 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:35:43.986156 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:35:43.987108 (+ 952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:35:43.987115 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:35:43.987150 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:35:43.987709 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:43.987724 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:43.989454 (+ 1730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:43.989459 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:43.991740 (+ 2281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:43.991760 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:43.993439 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:43.993444 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:43.993616 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:43.993700 (+ 84us) client_negotiation.cc:769] Sending connection context
0620 23:35:43.993842 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 23:35:43.994040 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2426,"thread_start_us":784,"threads_started":1}
I20260620 23:35:43.995137 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:43.983924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37478 (local address 127.0.97.2:33641)
0620 23:35:43.985225 (+ 1301us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:43.985237 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:43.986054 (+ 817us) server_negotiation.cc:407] Connection header received
0620 23:35:43.986363 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:43.986376 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:43.986630 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:43.986819 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:35:43.987903 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:43.989288 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:43.991928 (+ 2640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:43.993245 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:43.993396 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:43.993609 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:43.994710 (+ 1101us) server_negotiation.cc:299] Negotiation successful
0620 23:35:43.994858 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1091,"thread_start_us":953,"threads_started":1}
I20260620 23:35:43.995201 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:43.982834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37077 (local address 127.0.0.1:44508)
0620 23:35:43.984303 (+ 1469us) negotiation.cc:107] Waiting for socket to connect
0620 23:35:43.984357 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 23:35:43.984564 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:35:43.986460 (+ 1896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:35:43.986467 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:35:43.986500 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:35:43.986971 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:43.986984 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:43.989117 (+ 2133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:43.989122 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:43.992393 (+ 3271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:43.992407 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:43.994340 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:43.994346 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:43.994543 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:43.994594 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 23:35:43.994665 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 23:35:43.994799 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1115,"mutex_wait_us":1065,"thread_start_us":2614,"threads_started":1}
I20260620 23:35:43.996286 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:43.983104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44508 (local address 127.0.97.3:37077)
0620 23:35:43.985759 (+ 2655us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:43.985768 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:43.985801 (+ 33us) server_negotiation.cc:407] Connection header received
0620 23:35:43.985907 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:43.985916 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:43.986213 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:43.987195 (+ 982us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:35:43.987432 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:43.988959 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:43.992572 (+ 3613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:43.994112 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:43.995517 (+ 1405us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:43.995742 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:43.995806 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 23:35:43.995960 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2480,"thread_start_us":2319,"threads_started":1}
I20260620 23:35:43.997606 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:43.986709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40819 (local address 127.0.0.1:47690)
0620 23:35:43.988279 (+ 1570us) negotiation.cc:107] Waiting for socket to connect
0620 23:35:43.988293 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:35:43.988424 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:35:43.988928 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:35:43.988933 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:35:43.988955 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:35:43.989447 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:43.989468 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:43.991038 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:43.991045 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:43.993161 (+ 2116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:43.993174 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:43.996783 (+ 3609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:43.996790 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:43.996966 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:43.996998 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 23:35:43.997117 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 23:35:43.997240 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1470,"mutex_wait_us":30,"thread_start_us":1346,"threads_started":1}
I20260620 23:35:43.997602 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:43.986182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47690 (local address 127.0.97.1:40819)
0620 23:35:43.987062 (+ 880us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:43.987073 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:43.988393 (+ 1320us) server_negotiation.cc:407] Connection header received
0620 23:35:43.988581 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:43.988588 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:43.988760 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:43.988944 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:35:43.989684 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:43.990870 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:43.993350 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:43.996591 (+ 3241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:43.996739 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:43.996933 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:43.997154 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 23:35:43.997308 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":697,"thread_start_us":654,"threads_started":1}
I20260620 23:35:43.998842 2415 tablet_service.cc:1467] Processing CreateTablet for tablet 2cabd8969ea749c0a532055bef9877ef (DEFAULT_TABLE table=test-table [id=1fa3bc81b1954cf5b78be7d7c3091b1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:35:44.000370 2415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cabd8969ea749c0a532055bef9877ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:35:44.000696 2276 tablet_service.cc:1467] Processing CreateTablet for tablet 2cabd8969ea749c0a532055bef9877ef (DEFAULT_TABLE table=test-table [id=1fa3bc81b1954cf5b78be7d7c3091b1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:35:44.001144 2554 tablet_service.cc:1467] Processing CreateTablet for tablet 2cabd8969ea749c0a532055bef9877ef (DEFAULT_TABLE table=test-table [id=1fa3bc81b1954cf5b78be7d7c3091b1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:35:44.001786 2276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cabd8969ea749c0a532055bef9877ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:35:44.002394 2554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cabd8969ea749c0a532055bef9877ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:35:44.012233 2643 tablet_bootstrap.cc:492] T 2cabd8969ea749c0a532055bef9877ef P 1fcac2741ad84a55bc68e5dd99ae3bd5: Bootstrap starting.
I20260620 23:35:44.015057 2644 tablet_bootstrap.cc:492] T 2cabd8969ea749c0a532055bef9877ef P f10804aa6b1d4b22a5a82fb8e2e9d0d6: Bootstrap starting.
I20260620 23:35:44.016549 2645 tablet_bootstrap.cc:492] T 2cabd8969ea749c0a532055bef9877ef P fa4063ffe74c4ef183d404cf1d2c20cd: Bootstrap starting.
I20260620 23:35:44.017568 2643 tablet_bootstrap.cc:654] T 2cabd8969ea749c0a532055bef9877ef P 1fcac2741ad84a55bc68e5dd99ae3bd5: Neither blocks nor log segments found. Creating new log.
I20260620 23:35:44.018407 2644 tablet_bootstrap.cc:654] T 2cabd8969ea749c0a532055bef9877ef P f10804aa6b1d4b22a5a82fb8e2e9d0d6: Neither blocks nor log segments found. Creating new log.
I20260620 23:35:44.018977 2643 log.cc:824] T 2cabd8969ea749c0a532055bef9877ef P 1fcac2741ad84a55bc68e5dd99ae3bd5: Log is configured to *not* fsync() on all Append() calls
I20260620 23:35:44.019631 2645 tablet_bootstrap.cc:654] T 2cabd8969ea749c0a532055bef9877ef P fa4063ffe74c4ef183d404cf1d2c20cd: Neither blocks nor log segments found. Creating new log.
I20260620 23:35:44.019703 2644 log.cc:824] T 2cabd8969ea749c0a532055bef9877ef P f10804aa6b1d4b22a5a82fb8e2e9d0d6: Log is configured to *not* fsync() on all Append() calls
I20260620 23:35:44.020820 2645 log.cc:824] T 2cabd8969ea749c0a532055bef9877ef P fa4063ffe74c4ef183d404cf1d2c20cd: Log is configured to *not* fsync() on all Append() calls
I20260620 23:35:44.023132 2643 tablet_bootstrap.cc:492] T 2cabd8969ea749c0a532055bef9877ef P 1fcac2741ad84a55bc68e5dd99ae3bd5: No bootstrap required, opened a new log
I20260620 23:35:44.023499 2643 ts_tablet_manager.cc:1397] T 2cabd8969ea749c0a532055bef9877ef P 1fcac2741ad84a55bc68e5dd99ae3bd5: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20260620 23:35:44.026296 2644 tablet_bootstrap.cc:492] T 2cabd8969ea749c0a532055bef9877ef P f10804aa6b1d4b22a5a82fb8e2e9d0d6: No bootstrap required, opened a new log
I20260620 23:35:44.026418 2645 tablet_bootstrap.cc:492] T 2cabd8969ea749c0a532055bef9877ef P fa4063ffe74c4ef183d404cf1d2c20cd: No bootstrap required, opened a new log
I20260620 23:35:44.026587 2644 ts_tablet_manager.cc:1397] T 2cabd8969ea749c0a532055bef9877ef P f10804aa6b1d4b22a5a82fb8e2e9d0d6: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260620 23:35:44.026690 2645 ts_tablet_manager.cc:1397] T 2cabd8969ea749c0a532055bef9877ef P fa4063ffe74c4ef183d404cf1d2c20cd: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260620 23:35:44.033955 2643 raft_consensus.cc:348] T 2cabd8969ea749c0a532055bef9877ef P 1fcac2741ad84a55bc68e5dd99ae3bd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37077 } } peers { permanent_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40819 } } peers { permanent_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33641 } }
I20260620 23:35:44.034447 2643 raft_consensus.cc:374] T 2cabd8969ea749c0a532055bef9877ef P 1fcac2741ad84a55bc68e5dd99ae3bd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:35:44.034214 2645 raft_consensus.cc:348] T 2cabd8969ea749c0a532055bef9877ef P fa4063ffe74c4ef183d404cf1d2c20cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37077 } } peers { permanent_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40819 } } peers { permanent_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33641 } }
I20260620 23:35:44.034554 2643 raft_consensus.cc:729] T 2cabd8969ea749c0a532055bef9877ef P 1fcac2741ad84a55bc68e5dd99ae3bd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fcac2741ad84a55bc68e5dd99ae3bd5, State: Initialized, Role: FOLLOWER
I20260620 23:35:44.034592 2645 raft_consensus.cc:374] T 2cabd8969ea749c0a532055bef9877ef P fa4063ffe74c4ef183d404cf1d2c20cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:35:44.034701 2645 raft_consensus.cc:729] T 2cabd8969ea749c0a532055bef9877ef P fa4063ffe74c4ef183d404cf1d2c20cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa4063ffe74c4ef183d404cf1d2c20cd, State: Initialized, Role: FOLLOWER
I20260620 23:35:44.035300 2645 consensus_queue.cc:260] T 2cabd8969ea749c0a532055bef9877ef P fa4063ffe74c4ef183d404cf1d2c20cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37077 } } peers { permanent_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40819 } } peers { permanent_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33641 } }
I20260620 23:35:44.035318 2643 consensus_queue.cc:260] T 2cabd8969ea749c0a532055bef9877ef P 1fcac2741ad84a55bc68e5dd99ae3bd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37077 } } peers { permanent_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40819 } } peers { permanent_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33641 } }
I20260620 23:35:44.036516 2624 heartbeater.cc:503] Master 127.0.97.62:43025 was elected leader, sending a full tablet report...
I20260620 23:35:44.037216 2645 ts_tablet_manager.cc:1428] T 2cabd8969ea749c0a532055bef9877ef P fa4063ffe74c4ef183d404cf1d2c20cd: Time spent starting tablet: real 0.010s user 0.007s sys 0.004s
I20260620 23:35:44.037590 2644 raft_consensus.cc:348] T 2cabd8969ea749c0a532055bef9877ef P f10804aa6b1d4b22a5a82fb8e2e9d0d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37077 } } peers { permanent_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40819 } } peers { permanent_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33641 } }
I20260620 23:35:44.038133 2644 raft_consensus.cc:374] T 2cabd8969ea749c0a532055bef9877ef P f10804aa6b1d4b22a5a82fb8e2e9d0d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:35:44.038254 2644 raft_consensus.cc:729] T 2cabd8969ea749c0a532055bef9877ef P f10804aa6b1d4b22a5a82fb8e2e9d0d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f10804aa6b1d4b22a5a82fb8e2e9d0d6, State: Initialized, Role: FOLLOWER
I20260620 23:35:44.039018 2644 consensus_queue.cc:260] T 2cabd8969ea749c0a532055bef9877ef P f10804aa6b1d4b22a5a82fb8e2e9d0d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37077 } } peers { permanent_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40819 } } peers { permanent_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33641 } }
I20260620 23:35:44.041050 2644 ts_tablet_manager.cc:1428] T 2cabd8969ea749c0a532055bef9877ef P f10804aa6b1d4b22a5a82fb8e2e9d0d6: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260620 23:35:44.041589 2485 heartbeater.cc:503] Master 127.0.97.62:43025 was elected leader, sending a full tablet report...
I20260620 23:35:44.043510 2643 ts_tablet_manager.cc:1428] T 2cabd8969ea749c0a532055bef9877ef P 1fcac2741ad84a55bc68e5dd99ae3bd5: Time spent starting tablet: real 0.020s user 0.017s sys 0.001s
W20260620 23:35:44.088339 2347 tablet.cc:2369] T 2cabd8969ea749c0a532055bef9877ef P 1fcac2741ad84a55bc68e5dd99ae3bd5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:35:44.098248 2486 tablet.cc:2369] T 2cabd8969ea749c0a532055bef9877ef P f10804aa6b1d4b22a5a82fb8e2e9d0d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:35:44.148861 2625 tablet.cc:2369] T 2cabd8969ea749c0a532055bef9877ef P fa4063ffe74c4ef183d404cf1d2c20cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:35:44.209975 2651 raft_consensus.cc:482] T 2cabd8969ea749c0a532055bef9877ef P f10804aa6b1d4b22a5a82fb8e2e9d0d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:35:44.210350 2651 raft_consensus.cc:504] T 2cabd8969ea749c0a532055bef9877ef P f10804aa6b1d4b22a5a82fb8e2e9d0d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37077 } } peers { permanent_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40819 } } peers { permanent_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33641 } }
I20260620 23:35:44.212356 2651 leader_election.cc:288] T 2cabd8969ea749c0a532055bef9877ef P f10804aa6b1d4b22a5a82fb8e2e9d0d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa4063ffe74c4ef183d404cf1d2c20cd (127.0.97.3:37077), 1fcac2741ad84a55bc68e5dd99ae3bd5 (127.0.97.1:40819)
I20260620 23:35:44.214891 2650 raft_consensus.cc:482] T 2cabd8969ea749c0a532055bef9877ef P 1fcac2741ad84a55bc68e5dd99ae3bd5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:35:44.215237 2650 raft_consensus.cc:504] T 2cabd8969ea749c0a532055bef9877ef P 1fcac2741ad84a55bc68e5dd99ae3bd5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37077 } } peers { permanent_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40819 } } peers { permanent_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33641 } }
I20260620 23:35:44.222458 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:44.213058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37555 (local address 127.0.97.3:37077)
0620 23:35:44.213322 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:44.213332 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:44.214125 (+ 793us) server_negotiation.cc:407] Connection header received
0620 23:35:44.214206 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:44.214212 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:44.214302 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:44.214397 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:35:44.215487 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:44.216495 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:44.219471 (+ 2976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:44.220911 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:44.221052 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:44.221201 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:44.221910 (+ 709us) server_negotiation.cc:299] Negotiation successful
0620 23:35:44.222108 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 23:35:44.222721 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:44.214386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54467 (local address 127.0.97.1:40819)
0620 23:35:44.214748 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:44.214755 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:44.215036 (+ 281us) server_negotiation.cc:407] Connection header received
0620 23:35:44.215270 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:44.215276 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:44.215358 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:44.215422 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:35:44.216123 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:44.217207 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:44.220188 (+ 2981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:44.221447 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:44.221560 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:44.221652 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:44.222340 (+ 688us) server_negotiation.cc:299] Negotiation successful
0620 23:35:44.222480 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237}
I20260620 23:35:44.223418 2653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:44.214086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40819 (local address 127.0.97.2:54467)
0620 23:35:44.214917 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0620 23:35:44.214959 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 23:35:44.215128 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:35:44.215608 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:35:44.215614 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:35:44.215638 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:35:44.215969 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:44.215979 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:44.217418 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:44.217425 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:44.220009 (+ 2584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:44.220022 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:44.221970 (+ 1948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:44.221976 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:44.222139 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:44.222168 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 23:35:44.222854 (+ 686us) client_negotiation.cc:240] Negotiation successful
0620 23:35:44.223012 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"mutex_wait_us":416,"thread_start_us":535,"threads_started":1}
I20260620 23:35:44.223480 2652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:44.212963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37077 (local address 127.0.97.2:37555)
0620 23:35:44.213918 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0620 23:35:44.213940 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 23:35:44.214085 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:35:44.214842 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:35:44.214847 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:35:44.214881 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:35:44.215303 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:44.215315 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:44.216667 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:44.216672 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:44.219208 (+ 2536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:44.219222 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:44.221419 (+ 2197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:44.221427 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:44.221681 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:44.221734 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 23:35:44.222715 (+ 981us) client_negotiation.cc:240] Negotiation successful
0620 23:35:44.222867 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":774,"mutex_wait_us":1,"thread_start_us":578,"threads_started":1}
I20260620 23:35:44.223699 2650 leader_election.cc:288] T 2cabd8969ea749c0a532055bef9877ef P 1fcac2741ad84a55bc68e5dd99ae3bd5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa4063ffe74c4ef183d404cf1d2c20cd (127.0.97.3:37077), f10804aa6b1d4b22a5a82fb8e2e9d0d6 (127.0.97.2:33641)
I20260620 23:35:44.224572 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cabd8969ea749c0a532055bef9877ef" candidate_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" is_pre_election: true
I20260620 23:35:44.225041 2296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cabd8969ea749c0a532055bef9877ef" candidate_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" is_pre_election: true
I20260620 23:35:44.225195 2574 raft_consensus.cc:2454] T 2cabd8969ea749c0a532055bef9877ef P fa4063ffe74c4ef183d404cf1d2c20cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f10804aa6b1d4b22a5a82fb8e2e9d0d6 in term 0.
I20260620 23:35:44.225485 2296 raft_consensus.cc:2454] T 2cabd8969ea749c0a532055bef9877ef P 1fcac2741ad84a55bc68e5dd99ae3bd5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f10804aa6b1d4b22a5a82fb8e2e9d0d6 in term 0.
I20260620 23:35:44.225945 2366 leader_election.cc:302] T 2cabd8969ea749c0a532055bef9877ef P f10804aa6b1d4b22a5a82fb8e2e9d0d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f10804aa6b1d4b22a5a82fb8e2e9d0d6, fa4063ffe74c4ef183d404cf1d2c20cd; no voters:
I20260620 23:35:44.226543 2651 raft_consensus.cc:2781] T 2cabd8969ea749c0a532055bef9877ef P f10804aa6b1d4b22a5a82fb8e2e9d0d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:35:44.226686 2651 raft_consensus.cc:482] T 2cabd8969ea749c0a532055bef9877ef P f10804aa6b1d4b22a5a82fb8e2e9d0d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:35:44.226773 2651 raft_consensus.cc:3037] T 2cabd8969ea749c0a532055bef9877ef P f10804aa6b1d4b22a5a82fb8e2e9d0d6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:35:44.229933 2651 raft_consensus.cc:504] T 2cabd8969ea749c0a532055bef9877ef P f10804aa6b1d4b22a5a82fb8e2e9d0d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37077 } } peers { permanent_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40819 } } peers { permanent_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33641 } }
I20260620 23:35:44.230639 2651 leader_election.cc:288] T 2cabd8969ea749c0a532055bef9877ef P f10804aa6b1d4b22a5a82fb8e2e9d0d6 [CANDIDATE]: Term 1 election: Requested vote from peers fa4063ffe74c4ef183d404cf1d2c20cd (127.0.97.3:37077), 1fcac2741ad84a55bc68e5dd99ae3bd5 (127.0.97.1:40819)
I20260620 23:35:44.231153 2296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cabd8969ea749c0a532055bef9877ef" candidate_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5"
I20260620 23:35:44.231154 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cabd8969ea749c0a532055bef9877ef" candidate_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd"
I20260620 23:35:44.231359 2296 raft_consensus.cc:3037] T 2cabd8969ea749c0a532055bef9877ef P 1fcac2741ad84a55bc68e5dd99ae3bd5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:35:44.231365 2574 raft_consensus.cc:3037] T 2cabd8969ea749c0a532055bef9877ef P fa4063ffe74c4ef183d404cf1d2c20cd [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:35:44.233431 2656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:44.217592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37077 (local address 127.0.97.1:32975)
0620 23:35:44.224532 (+ 6940us) negotiation.cc:107] Waiting for socket to connect
0620 23:35:44.224553 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:35:44.224687 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:35:44.225293 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:35:44.225298 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:35:44.225321 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:35:44.225682 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:44.225692 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:44.228090 (+ 2398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:44.228096 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:44.230462 (+ 2366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:44.230477 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:44.232475 (+ 1998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:44.232483 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:44.232662 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:44.232699 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 23:35:44.232786 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 23:35:44.232961 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6777,"mutex_wait_us":1,"thread_start_us":6402,"threads_started":1}
I20260620 23:35:44.233799 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:44.220213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47485 (local address 127.0.97.2:33641)
0620 23:35:44.221812 (+ 1599us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:44.221817 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:44.226454 (+ 4637us) server_negotiation.cc:407] Connection header received
0620 23:35:44.226786 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:44.226791 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:44.226866 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:44.226928 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:35:44.227747 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:44.228927 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:44.231372 (+ 2445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:44.232572 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:44.232672 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:44.232778 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:44.233396 (+ 618us) server_negotiation.cc:299] Negotiation successful
0620 23:35:44.233551 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1465}
I20260620 23:35:44.234154 2655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:44.218227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:32975 (local address 127.0.97.3:37077)
0620 23:35:44.219444 (+ 1217us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:44.219449 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:44.224800 (+ 5351us) server_negotiation.cc:407] Connection header received
0620 23:35:44.224863 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:44.224869 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:44.224932 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:44.225009 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:35:44.225860 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:44.227919 (+ 2059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:44.230644 (+ 2725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:44.232060 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:44.232182 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:44.232322 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:44.233626 (+ 1304us) server_negotiation.cc:299] Negotiation successful
0620 23:35:44.233761 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1078,"thread_start_us":938,"threads_started":1}
I20260620 23:35:44.234722 2296 raft_consensus.cc:2454] T 2cabd8969ea749c0a532055bef9877ef P 1fcac2741ad84a55bc68e5dd99ae3bd5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f10804aa6b1d4b22a5a82fb8e2e9d0d6 in term 1.
I20260620 23:35:44.234783 2573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cabd8969ea749c0a532055bef9877ef" candidate_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" is_pre_election: true
I20260620 23:35:44.235134 2654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:44.219820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33641 (local address 127.0.97.1:47485)
0620 23:35:44.226368 (+ 6548us) negotiation.cc:107] Waiting for socket to connect
0620 23:35:44.226385 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:35:44.226552 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:35:44.227210 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:35:44.227214 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:35:44.227240 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:35:44.227580 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:44.227591 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:44.229126 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:44.229131 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:44.231190 (+ 2059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:44.231204 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:44.232970 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:44.232979 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:44.233183 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:44.233216 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 23:35:44.234163 (+ 947us) client_negotiation.cc:240] Negotiation successful
0620 23:35:44.234674 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6436,"mutex_wait_us":25,"thread_start_us":4394,"threads_started":1}
I20260620 23:35:44.235183 2366 leader_election.cc:302] T 2cabd8969ea749c0a532055bef9877ef P f10804aa6b1d4b22a5a82fb8e2e9d0d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fcac2741ad84a55bc68e5dd99ae3bd5, f10804aa6b1d4b22a5a82fb8e2e9d0d6; no voters:
I20260620 23:35:44.235530 2651 raft_consensus.cc:2781] T 2cabd8969ea749c0a532055bef9877ef P f10804aa6b1d4b22a5a82fb8e2e9d0d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:35:44.235525 2574 raft_consensus.cc:2454] T 2cabd8969ea749c0a532055bef9877ef P fa4063ffe74c4ef183d404cf1d2c20cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f10804aa6b1d4b22a5a82fb8e2e9d0d6 in term 1.
I20260620 23:35:44.236085 2435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cabd8969ea749c0a532055bef9877ef" candidate_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" is_pre_election: true
I20260620 23:35:44.236398 2651 raft_consensus.cc:686] T 2cabd8969ea749c0a532055bef9877ef P f10804aa6b1d4b22a5a82fb8e2e9d0d6 [term 1 LEADER]: Becoming Leader. State: Replica: f10804aa6b1d4b22a5a82fb8e2e9d0d6, State: Running, Role: LEADER
I20260620 23:35:44.236982 2651 consensus_queue.cc:237] T 2cabd8969ea749c0a532055bef9877ef P f10804aa6b1d4b22a5a82fb8e2e9d0d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37077 } } peers { permanent_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40819 } } peers { permanent_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33641 } }
I20260620 23:35:44.241329 2229 leader_election.cc:302] T 2cabd8969ea749c0a532055bef9877ef P 1fcac2741ad84a55bc68e5dd99ae3bd5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fcac2741ad84a55bc68e5dd99ae3bd5; no voters: f10804aa6b1d4b22a5a82fb8e2e9d0d6, fa4063ffe74c4ef183d404cf1d2c20cd
I20260620 23:35:44.241768 2650 raft_consensus.cc:2726] T 2cabd8969ea749c0a532055bef9877ef P 1fcac2741ad84a55bc68e5dd99ae3bd5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 23:35:44.242317 2139 catalog_manager.cc:5495] T 2cabd8969ea749c0a532055bef9877ef P f10804aa6b1d4b22a5a82fb8e2e9d0d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to f10804aa6b1d4b22a5a82fb8e2e9d0d6 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33641 } health_report { overall_health: HEALTHY } } }
Jun 20 23:35:44 dist-test-slave-tcgx krb5kdc[2094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998542, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:35:44.326565 2664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:44.313431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43025 (local address 127.0.0.1:37194)
0620 23:35:44.314126 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0620 23:35:44.314143 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:35:44.314278 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:35:44.314582 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:35:44.314585 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:35:44.315387 (+ 802us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:35:44.315656 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:44.315664 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:44.316617 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:44.316620 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:44.317691 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:44.317701 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:44.318442 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:44.318447 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:44.318572 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:44.319273 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:35:44.319291 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:35:44.321514 (+ 2223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:35:44.324397 (+ 2883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:35:44.324407 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:35:44.324414 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:35:44.324819 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:35:44.325134 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:35:44.325137 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:35:44.325139 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:35:44.325285 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:35:44.325713 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:35:44.325719 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:35:44.325860 (+ 141us) client_negotiation.cc:769] Sending connection context
0620 23:35:44.325993 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 23:35:44.326192 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":385,"threads_started":1}
I20260620 23:35:44.326567 2234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:44.313417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37194 (local address 127.0.97.62:43025)
0620 23:35:44.313661 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:44.313670 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:44.314239 (+ 569us) server_negotiation.cc:407] Connection header received
0620 23:35:44.314384 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:44.314387 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:44.314446 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:44.314528 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:35:44.315795 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:44.316496 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:44.317805 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:44.318246 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:44.318340 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:44.321693 (+ 3353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:35:44.321698 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:35:44.321701 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:35:44.321728 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:35:44.324280 (+ 2552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:44.324918 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:44.324922 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:44.324924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:44.325041 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:44.325375 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:44.325378 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:44.325380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:44.325605 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:35:44.325715 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:44.326079 (+ 364us) server_negotiation.cc:299] Negotiation successful
0620 23:35:44.326312 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":79}
I20260620 23:35:44.331554 2139 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37194:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 23:35:44.332594 2139 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:35:44.339646 2554 tablet_service.cc:1467] Processing CreateTablet for tablet ddc29ab67d7a4ed48235e8049b3f9ad7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0e9cac3d95224c878094a092d5694572]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 23:35:44.339416 2415 tablet_service.cc:1467] Processing CreateTablet for tablet ddc29ab67d7a4ed48235e8049b3f9ad7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0e9cac3d95224c878094a092d5694572]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 23:35:44.339341 2276 tablet_service.cc:1467] Processing CreateTablet for tablet ddc29ab67d7a4ed48235e8049b3f9ad7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0e9cac3d95224c878094a092d5694572]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 23:35:44.340402 2276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddc29ab67d7a4ed48235e8049b3f9ad7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:35:44.340417 2415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddc29ab67d7a4ed48235e8049b3f9ad7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:35:44.340399 2554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddc29ab67d7a4ed48235e8049b3f9ad7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:35:44.346444 2645 tablet_bootstrap.cc:492] T ddc29ab67d7a4ed48235e8049b3f9ad7 P fa4063ffe74c4ef183d404cf1d2c20cd: Bootstrap starting.
I20260620 23:35:44.348070 2645 tablet_bootstrap.cc:654] T ddc29ab67d7a4ed48235e8049b3f9ad7 P fa4063ffe74c4ef183d404cf1d2c20cd: Neither blocks nor log segments found. Creating new log.
I20260620 23:35:44.348199 2644 tablet_bootstrap.cc:492] T ddc29ab67d7a4ed48235e8049b3f9ad7 P f10804aa6b1d4b22a5a82fb8e2e9d0d6: Bootstrap starting.
I20260620 23:35:44.349098 2643 tablet_bootstrap.cc:492] T ddc29ab67d7a4ed48235e8049b3f9ad7 P 1fcac2741ad84a55bc68e5dd99ae3bd5: Bootstrap starting.
I20260620 23:35:44.349502 2644 tablet_bootstrap.cc:654] T ddc29ab67d7a4ed48235e8049b3f9ad7 P f10804aa6b1d4b22a5a82fb8e2e9d0d6: Neither blocks nor log segments found. Creating new log.
I20260620 23:35:44.350584 2643 tablet_bootstrap.cc:654] T ddc29ab67d7a4ed48235e8049b3f9ad7 P 1fcac2741ad84a55bc68e5dd99ae3bd5: Neither blocks nor log segments found. Creating new log.
I20260620 23:35:44.353268 2643 tablet_bootstrap.cc:492] T ddc29ab67d7a4ed48235e8049b3f9ad7 P 1fcac2741ad84a55bc68e5dd99ae3bd5: No bootstrap required, opened a new log
I20260620 23:35:44.353431 2643 ts_tablet_manager.cc:1397] T ddc29ab67d7a4ed48235e8049b3f9ad7 P 1fcac2741ad84a55bc68e5dd99ae3bd5: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260620 23:35:44.354423 2643 raft_consensus.cc:348] T ddc29ab67d7a4ed48235e8049b3f9ad7 P 1fcac2741ad84a55bc68e5dd99ae3bd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40819 } } peers { permanent_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33641 } } peers { permanent_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37077 } }
I20260620 23:35:44.354756 2643 raft_consensus.cc:374] T ddc29ab67d7a4ed48235e8049b3f9ad7 P 1fcac2741ad84a55bc68e5dd99ae3bd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:35:44.354858 2643 raft_consensus.cc:729] T ddc29ab67d7a4ed48235e8049b3f9ad7 P 1fcac2741ad84a55bc68e5dd99ae3bd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fcac2741ad84a55bc68e5dd99ae3bd5, State: Initialized, Role: FOLLOWER
I20260620 23:35:44.356714 2643 consensus_queue.cc:260] T ddc29ab67d7a4ed48235e8049b3f9ad7 P 1fcac2741ad84a55bc68e5dd99ae3bd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40819 } } peers { permanent_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33641 } } peers { permanent_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37077 } }
I20260620 23:35:44.357326 2650 tablet_replica.cc:439] T ddc29ab67d7a4ed48235e8049b3f9ad7 P 1fcac2741ad84a55bc68e5dd99ae3bd5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40819 } } peers { permanent_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33641 } } peers { permanent_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37077 } } }
I20260620 23:35:44.357607 2643 ts_tablet_manager.cc:1428] T ddc29ab67d7a4ed48235e8049b3f9ad7 P 1fcac2741ad84a55bc68e5dd99ae3bd5: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260620 23:35:44.357613 2650 tablet_replica.cc:442] T ddc29ab67d7a4ed48235e8049b3f9ad7 P 1fcac2741ad84a55bc68e5dd99ae3bd5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:35:44.361567 2645 tablet_bootstrap.cc:492] T ddc29ab67d7a4ed48235e8049b3f9ad7 P fa4063ffe74c4ef183d404cf1d2c20cd: No bootstrap required, opened a new log
I20260620 23:35:44.361747 2645 ts_tablet_manager.cc:1397] T ddc29ab67d7a4ed48235e8049b3f9ad7 P fa4063ffe74c4ef183d404cf1d2c20cd: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.003s
I20260620 23:35:44.362030 2644 tablet_bootstrap.cc:492] T ddc29ab67d7a4ed48235e8049b3f9ad7 P f10804aa6b1d4b22a5a82fb8e2e9d0d6: No bootstrap required, opened a new log
I20260620 23:35:44.362232 2644 ts_tablet_manager.cc:1397] T ddc29ab67d7a4ed48235e8049b3f9ad7 P f10804aa6b1d4b22a5a82fb8e2e9d0d6: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20260620 23:35:44.362576 2645 raft_consensus.cc:348] T ddc29ab67d7a4ed48235e8049b3f9ad7 P fa4063ffe74c4ef183d404cf1d2c20cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40819 } } peers { permanent_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33641 } } peers { permanent_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37077 } }
I20260620 23:35:44.362829 2645 raft_consensus.cc:374] T ddc29ab67d7a4ed48235e8049b3f9ad7 P fa4063ffe74c4ef183d404cf1d2c20cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:35:44.362959 2645 raft_consensus.cc:729] T ddc29ab67d7a4ed48235e8049b3f9ad7 P fa4063ffe74c4ef183d404cf1d2c20cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa4063ffe74c4ef183d404cf1d2c20cd, State: Initialized, Role: FOLLOWER
I20260620 23:35:44.363080 2644 raft_consensus.cc:348] T ddc29ab67d7a4ed48235e8049b3f9ad7 P f10804aa6b1d4b22a5a82fb8e2e9d0d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40819 } } peers { permanent_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33641 } } peers { permanent_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37077 } }
I20260620 23:35:44.363337 2644 raft_consensus.cc:374] T ddc29ab67d7a4ed48235e8049b3f9ad7 P f10804aa6b1d4b22a5a82fb8e2e9d0d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:35:44.363209 2645 consensus_queue.cc:260] T ddc29ab67d7a4ed48235e8049b3f9ad7 P fa4063ffe74c4ef183d404cf1d2c20cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40819 } } peers { permanent_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33641 } } peers { permanent_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37077 } }
I20260620 23:35:44.363431 2644 raft_consensus.cc:729] T ddc29ab67d7a4ed48235e8049b3f9ad7 P f10804aa6b1d4b22a5a82fb8e2e9d0d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f10804aa6b1d4b22a5a82fb8e2e9d0d6, State: Initialized, Role: FOLLOWER
I20260620 23:35:44.363652 2644 consensus_queue.cc:260] T ddc29ab67d7a4ed48235e8049b3f9ad7 P f10804aa6b1d4b22a5a82fb8e2e9d0d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40819 } } peers { permanent_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33641 } } peers { permanent_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37077 } }
I20260620 23:35:44.363832 2645 ts_tablet_manager.cc:1428] T ddc29ab67d7a4ed48235e8049b3f9ad7 P fa4063ffe74c4ef183d404cf1d2c20cd: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 23:35:44.363701 2649 tablet_replica.cc:439] T ddc29ab67d7a4ed48235e8049b3f9ad7 P fa4063ffe74c4ef183d404cf1d2c20cd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40819 } } peers { permanent_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33641 } } peers { permanent_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37077 } } }
I20260620 23:35:44.364068 2649 tablet_replica.cc:442] T ddc29ab67d7a4ed48235e8049b3f9ad7 P fa4063ffe74c4ef183d404cf1d2c20cd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:35:44.364099 2651 tablet_replica.cc:439] T ddc29ab67d7a4ed48235e8049b3f9ad7 P f10804aa6b1d4b22a5a82fb8e2e9d0d6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40819 } } peers { permanent_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33641 } } peers { permanent_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37077 } } }
I20260620 23:35:44.364370 2651 tablet_replica.cc:442] T ddc29ab67d7a4ed48235e8049b3f9ad7 P f10804aa6b1d4b22a5a82fb8e2e9d0d6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:35:44.364795 2644 ts_tablet_manager.cc:1428] T ddc29ab67d7a4ed48235e8049b3f9ad7 P f10804aa6b1d4b22a5a82fb8e2e9d0d6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:35:44.597240 2650 raft_consensus.cc:482] T ddc29ab67d7a4ed48235e8049b3f9ad7 P 1fcac2741ad84a55bc68e5dd99ae3bd5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:35:44.597427 2650 raft_consensus.cc:504] T ddc29ab67d7a4ed48235e8049b3f9ad7 P 1fcac2741ad84a55bc68e5dd99ae3bd5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40819 } } peers { permanent_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33641 } } peers { permanent_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37077 } }
I20260620 23:35:44.598023 2650 leader_election.cc:288] T ddc29ab67d7a4ed48235e8049b3f9ad7 P 1fcac2741ad84a55bc68e5dd99ae3bd5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f10804aa6b1d4b22a5a82fb8e2e9d0d6 (127.0.97.2:33641), fa4063ffe74c4ef183d404cf1d2c20cd (127.0.97.3:37077)
I20260620 23:35:44.598615 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc29ab67d7a4ed48235e8049b3f9ad7" candidate_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" is_pre_election: true
I20260620 23:35:44.598629 2435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc29ab67d7a4ed48235e8049b3f9ad7" candidate_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" is_pre_election: true
I20260620 23:35:44.598888 2574 raft_consensus.cc:2454] T ddc29ab67d7a4ed48235e8049b3f9ad7 P fa4063ffe74c4ef183d404cf1d2c20cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fcac2741ad84a55bc68e5dd99ae3bd5 in term 0.
I20260620 23:35:44.598944 2435 raft_consensus.cc:2454] T ddc29ab67d7a4ed48235e8049b3f9ad7 P f10804aa6b1d4b22a5a82fb8e2e9d0d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fcac2741ad84a55bc68e5dd99ae3bd5 in term 0.
I20260620 23:35:44.599434 2226 leader_election.cc:302] T ddc29ab67d7a4ed48235e8049b3f9ad7 P 1fcac2741ad84a55bc68e5dd99ae3bd5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fcac2741ad84a55bc68e5dd99ae3bd5, fa4063ffe74c4ef183d404cf1d2c20cd; no voters:
I20260620 23:35:44.599758 2650 raft_consensus.cc:2781] T ddc29ab67d7a4ed48235e8049b3f9ad7 P 1fcac2741ad84a55bc68e5dd99ae3bd5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:35:44.599884 2650 raft_consensus.cc:482] T ddc29ab67d7a4ed48235e8049b3f9ad7 P 1fcac2741ad84a55bc68e5dd99ae3bd5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:35:44.599965 2650 raft_consensus.cc:3037] T ddc29ab67d7a4ed48235e8049b3f9ad7 P 1fcac2741ad84a55bc68e5dd99ae3bd5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:35:44.602063 2650 raft_consensus.cc:504] T ddc29ab67d7a4ed48235e8049b3f9ad7 P 1fcac2741ad84a55bc68e5dd99ae3bd5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40819 } } peers { permanent_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33641 } } peers { permanent_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37077 } }
I20260620 23:35:44.602608 2650 leader_election.cc:288] T ddc29ab67d7a4ed48235e8049b3f9ad7 P 1fcac2741ad84a55bc68e5dd99ae3bd5 [CANDIDATE]: Term 1 election: Requested vote from peers f10804aa6b1d4b22a5a82fb8e2e9d0d6 (127.0.97.2:33641), fa4063ffe74c4ef183d404cf1d2c20cd (127.0.97.3:37077)
I20260620 23:35:44.603152 2435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc29ab67d7a4ed48235e8049b3f9ad7" candidate_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6"
I20260620 23:35:44.603190 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc29ab67d7a4ed48235e8049b3f9ad7" candidate_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd"
I20260620 23:35:44.603366 2435 raft_consensus.cc:3037] T ddc29ab67d7a4ed48235e8049b3f9ad7 P f10804aa6b1d4b22a5a82fb8e2e9d0d6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:35:44.603430 2574 raft_consensus.cc:3037] T ddc29ab67d7a4ed48235e8049b3f9ad7 P fa4063ffe74c4ef183d404cf1d2c20cd [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:35:44.605374 2574 raft_consensus.cc:2454] T ddc29ab67d7a4ed48235e8049b3f9ad7 P fa4063ffe74c4ef183d404cf1d2c20cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fcac2741ad84a55bc68e5dd99ae3bd5 in term 1.
I20260620 23:35:44.605458 2435 raft_consensus.cc:2454] T ddc29ab67d7a4ed48235e8049b3f9ad7 P f10804aa6b1d4b22a5a82fb8e2e9d0d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fcac2741ad84a55bc68e5dd99ae3bd5 in term 1.
I20260620 23:35:44.605887 2226 leader_election.cc:302] T ddc29ab67d7a4ed48235e8049b3f9ad7 P 1fcac2741ad84a55bc68e5dd99ae3bd5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fcac2741ad84a55bc68e5dd99ae3bd5, fa4063ffe74c4ef183d404cf1d2c20cd; no voters:
I20260620 23:35:44.606181 2650 raft_consensus.cc:2781] T ddc29ab67d7a4ed48235e8049b3f9ad7 P 1fcac2741ad84a55bc68e5dd99ae3bd5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:35:44.606596 2650 raft_consensus.cc:686] T ddc29ab67d7a4ed48235e8049b3f9ad7 P 1fcac2741ad84a55bc68e5dd99ae3bd5 [term 1 LEADER]: Becoming Leader. State: Replica: 1fcac2741ad84a55bc68e5dd99ae3bd5, State: Running, Role: LEADER
I20260620 23:35:44.607168 2650 consensus_queue.cc:237] T ddc29ab67d7a4ed48235e8049b3f9ad7 P 1fcac2741ad84a55bc68e5dd99ae3bd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40819 } } peers { permanent_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33641 } } peers { permanent_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37077 } }
I20260620 23:35:44.610636 2668 tablet_replica.cc:439] T ddc29ab67d7a4ed48235e8049b3f9ad7 P 1fcac2741ad84a55bc68e5dd99ae3bd5: TxnStatusTablet state changed. Reason: New leader 1fcac2741ad84a55bc68e5dd99ae3bd5. Latest consensus state: current_term: 1 leader_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40819 } } peers { permanent_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33641 } } peers { permanent_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37077 } } }
I20260620 23:35:44.611054 2668 tablet_replica.cc:442] T ddc29ab67d7a4ed48235e8049b3f9ad7 P 1fcac2741ad84a55bc68e5dd99ae3bd5: This TxnStatusTablet replica's current role is: LEADER
I20260620 23:35:44.611779 2670 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 23:35:44.612093 2670 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 23:35:44.612239 2137 catalog_manager.cc:5495] T ddc29ab67d7a4ed48235e8049b3f9ad7 P 1fcac2741ad84a55bc68e5dd99ae3bd5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1fcac2741ad84a55bc68e5dd99ae3bd5 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40819 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33641 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37077 } health_report { overall_health: UNKNOWN } } }
I20260620 23:35:44.621557 2651 consensus_queue.cc:1035] T 2cabd8969ea749c0a532055bef9877ef P f10804aa6b1d4b22a5a82fb8e2e9d0d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:35:44.628921 2651 consensus_queue.cc:1035] T 2cabd8969ea749c0a532055bef9877ef P f10804aa6b1d4b22a5a82fb8e2e9d0d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:35:44.662526 2664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:44.655967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40819 (local address 127.0.0.1:47704)
0620 23:35:44.656218 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 23:35:44.656242 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 23:35:44.656358 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:35:44.656883 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:35:44.656887 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:35:44.656897 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:35:44.657061 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:44.657069 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:44.658511 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:44.658515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:44.659721 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:44.659733 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:44.660757 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:44.660762 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:44.660897 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:44.660976 (+ 79us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:35:44.662097 (+ 1121us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:35:44.662113 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 23:35:44.662179 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 23:35:44.662297 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 23:35:44.662611 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:44.656125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47704 (local address 127.0.97.1:40819)
0620 23:35:44.656302 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:44.656309 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:44.656330 (+ 21us) server_negotiation.cc:407] Connection header received
0620 23:35:44.656577 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:44.656583 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:44.656689 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:44.656845 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:35:44.657191 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:44.658357 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:44.659863 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:44.660565 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:44.660713 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:44.661182 (+ 469us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:35:44.661957 (+ 775us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:35:44.662122 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:44.662207 (+ 85us) server_negotiation.cc:299] Negotiation successful
0620 23:35:44.662299 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 23:35:44.669624 2574 raft_consensus.cc:1261] T ddc29ab67d7a4ed48235e8049b3f9ad7 P fa4063ffe74c4ef183d404cf1d2c20cd [term 1 FOLLOWER]: Refusing update from remote peer 1fcac2741ad84a55bc68e5dd99ae3bd5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:35:44.669708 2435 raft_consensus.cc:1261] T ddc29ab67d7a4ed48235e8049b3f9ad7 P f10804aa6b1d4b22a5a82fb8e2e9d0d6 [term 1 FOLLOWER]: Refusing update from remote peer 1fcac2741ad84a55bc68e5dd99ae3bd5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:35:44.670372 2650 consensus_queue.cc:1035] T ddc29ab67d7a4ed48235e8049b3f9ad7 P 1fcac2741ad84a55bc68e5dd99ae3bd5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:35:44.670707 2668 consensus_queue.cc:1035] T ddc29ab67d7a4ed48235e8049b3f9ad7 P 1fcac2741ad84a55bc68e5dd99ae3bd5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:35:44.683887 2649 tablet_replica.cc:439] T ddc29ab67d7a4ed48235e8049b3f9ad7 P fa4063ffe74c4ef183d404cf1d2c20cd: TxnStatusTablet state changed. Reason: New leader 1fcac2741ad84a55bc68e5dd99ae3bd5. Latest consensus state: current_term: 1 leader_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40819 } } peers { permanent_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33641 } } peers { permanent_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37077 } } }
I20260620 23:35:44.684293 2649 tablet_replica.cc:442] T ddc29ab67d7a4ed48235e8049b3f9ad7 P fa4063ffe74c4ef183d404cf1d2c20cd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:35:44.687366 2668 tablet_replica.cc:439] T ddc29ab67d7a4ed48235e8049b3f9ad7 P 1fcac2741ad84a55bc68e5dd99ae3bd5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40819 } } peers { permanent_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33641 } } peers { permanent_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37077 } } }
I20260620 23:35:44.687659 2668 tablet_replica.cc:442] T ddc29ab67d7a4ed48235e8049b3f9ad7 P 1fcac2741ad84a55bc68e5dd99ae3bd5: This TxnStatusTablet replica's current role is: LEADER
I20260620 23:35:44.688892 2676 mvcc.cc:205] Tried to move back new op lower bound from 7299066038958739456 to 7299066038714875904. Current Snapshot: MvccSnapshot[applied={T|T < 7299066038958739456}]
I20260620 23:35:44.689412 2649 tablet_replica.cc:439] T ddc29ab67d7a4ed48235e8049b3f9ad7 P fa4063ffe74c4ef183d404cf1d2c20cd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40819 } } peers { permanent_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33641 } } peers { permanent_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37077 } } }
I20260620 23:35:44.689668 2649 tablet_replica.cc:442] T ddc29ab67d7a4ed48235e8049b3f9ad7 P fa4063ffe74c4ef183d404cf1d2c20cd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:35:44.692242 2674 tablet_replica.cc:439] T ddc29ab67d7a4ed48235e8049b3f9ad7 P f10804aa6b1d4b22a5a82fb8e2e9d0d6: TxnStatusTablet state changed. Reason: New leader 1fcac2741ad84a55bc68e5dd99ae3bd5. Latest consensus state: current_term: 1 leader_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40819 } } peers { permanent_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33641 } } peers { permanent_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37077 } } }
I20260620 23:35:44.692623 2674 tablet_replica.cc:442] T ddc29ab67d7a4ed48235e8049b3f9ad7 P f10804aa6b1d4b22a5a82fb8e2e9d0d6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:35:44.693441 2650 tablet_replica.cc:439] T ddc29ab67d7a4ed48235e8049b3f9ad7 P 1fcac2741ad84a55bc68e5dd99ae3bd5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40819 } } peers { permanent_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33641 } } peers { permanent_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37077 } } }
I20260620 23:35:44.693760 2650 tablet_replica.cc:442] T ddc29ab67d7a4ed48235e8049b3f9ad7 P 1fcac2741ad84a55bc68e5dd99ae3bd5: This TxnStatusTablet replica's current role is: LEADER
I20260620 23:35:44.696025 2673 mvcc.cc:205] Tried to move back new op lower bound from 7299066038958739456 to 7299066038714875904. Current Snapshot: MvccSnapshot[applied={T|T < 7299066038958739456}]
I20260620 23:35:44.698866 2674 tablet_replica.cc:439] T ddc29ab67d7a4ed48235e8049b3f9ad7 P f10804aa6b1d4b22a5a82fb8e2e9d0d6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40819 } } peers { permanent_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33641 } } peers { permanent_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37077 } } }
I20260620 23:35:44.699167 2674 tablet_replica.cc:442] T ddc29ab67d7a4ed48235e8049b3f9ad7 P f10804aa6b1d4b22a5a82fb8e2e9d0d6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:35:44.708648 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:44.702328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37482 (local address 127.0.97.2:33641)
0620 23:35:44.702475 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:44.702479 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:44.702520 (+ 41us) server_negotiation.cc:407] Connection header received
0620 23:35:44.702607 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:44.702613 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:44.702722 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:44.702838 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:35:44.703322 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:44.704051 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:44.706524 (+ 2473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:44.706911 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:44.707081 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:44.707502 (+ 421us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:35:44.708031 (+ 529us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:35:44.708155 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:44.708289 (+ 134us) server_negotiation.cc:299] Negotiation successful
0620 23:35:44.708354 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 23:35:44.729099 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:44.720094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36979 (local address 127.0.97.1:40819)
0620 23:35:44.720297 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:44.720306 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:44.720984 (+ 678us) server_negotiation.cc:407] Connection header received
0620 23:35:44.721271 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:44.721277 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:44.721402 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:44.721520 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:35:44.722262 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:44.723526 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:44.726501 (+ 2975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:44.727224 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:44.727364 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:44.728134 (+ 770us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:35:44.728281 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:35:44.728376 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:44.728597 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 23:35:44.728690 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 23:35:44.730026 2688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:44.720084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40819 (local address 127.0.97.2:36979)
0620 23:35:44.720862 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0620 23:35:44.720884 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 23:35:44.721044 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:35:44.721571 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:35:44.721580 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:35:44.721599 (+ 19us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:35:44.722098 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:44.722115 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:44.724019 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:44.724028 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:44.726314 (+ 2286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:44.726337 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:44.727397 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:44.727406 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:44.727652 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:44.727753 (+ 101us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:35:44.728431 (+ 678us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:35:44.728445 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 23:35:44.729525 (+ 1080us) client_negotiation.cc:240] Negotiation successful
0620 23:35:44.729693 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":385,"threads_started":1}
I20260620 23:35:44.773612 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:44.766631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60697 (local address 127.0.97.2:33641)
0620 23:35:44.766841 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:44.766847 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:44.767283 (+ 436us) server_negotiation.cc:407] Connection header received
0620 23:35:44.767447 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:44.767451 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:44.767521 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:44.767654 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:35:44.768117 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:44.768869 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:44.770838 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:44.771557 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:44.771684 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:44.772649 (+ 965us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:35:44.772786 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:35:44.772991 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:44.773219 (+ 228us) server_negotiation.cc:299] Negotiation successful
0620 23:35:44.773303 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 23:35:44.774328 2691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:44.766581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33641 (local address 127.0.97.1:60697)
0620 23:35:44.767189 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0620 23:35:44.767208 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:35:44.767345 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:35:44.767684 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:35:44.767694 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:35:44.767708 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:35:44.767971 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:44.767982 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:44.769003 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:44.769009 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:44.770681 (+ 1672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:44.770696 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:44.772148 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:44.772155 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:44.772361 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:44.772438 (+ 77us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:35:44.773088 (+ 650us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:35:44.773098 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 23:35:44.773789 (+ 691us) client_negotiation.cc:240] Negotiation successful
0620 23:35:44.773944 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":212,"threads_started":1}
I20260620 23:35:44.814740 2138 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37178:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:35:44.815074 2138 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37178:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:35:44.819258 2138 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 78b3f3598f0540858739235691fbe934: Sending DeleteTablet for 3 replicas of tablet 2cabd8969ea749c0a532055bef9877ef
I20260620 23:35:44.820899 2276 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cabd8969ea749c0a532055bef9877ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:35:44 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47690
I20260620 23:35:44.820899 2554 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cabd8969ea749c0a532055bef9877ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:35:44 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44508
I20260620 23:35:44.821734 2415 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cabd8969ea749c0a532055bef9877ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:35:44 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37478
I20260620 23:35:44.839548 2138 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37178:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:35:44.840314 2138 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:35:44.849308 2414 tablet_service.cc:1467] Processing CreateTablet for tablet 3c699dc2009040098a7e455f738f84b6 (DEFAULT_TABLE table=test-table [id=802cd0100ebf48f097ac82e39da5a451]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:35:44.849972 2414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c699dc2009040098a7e455f738f84b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:35:44.849995 2275 tablet_service.cc:1467] Processing CreateTablet for tablet 3c699dc2009040098a7e455f738f84b6 (DEFAULT_TABLE table=test-table [id=802cd0100ebf48f097ac82e39da5a451]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:35:44.850598 2275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c699dc2009040098a7e455f738f84b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:35:44.854127 2553 tablet_service.cc:1467] Processing CreateTablet for tablet 3c699dc2009040098a7e455f738f84b6 (DEFAULT_TABLE table=test-table [id=802cd0100ebf48f097ac82e39da5a451]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:35:44.854660 2553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c699dc2009040098a7e455f738f84b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:35:44.833374 2695 tablet_replica.cc:330] T 2cabd8969ea749c0a532055bef9877ef P 1fcac2741ad84a55bc68e5dd99ae3bd5: stopping tablet replica
I20260620 23:35:44.872440 2695 raft_consensus.cc:2229] T 2cabd8969ea749c0a532055bef9877ef P 1fcac2741ad84a55bc68e5dd99ae3bd5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:35:44.872897 2695 raft_consensus.cc:2258] T 2cabd8969ea749c0a532055bef9877ef P 1fcac2741ad84a55bc68e5dd99ae3bd5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:35:44.875355 2699 tablet_bootstrap.cc:492] T 3c699dc2009040098a7e455f738f84b6 P fa4063ffe74c4ef183d404cf1d2c20cd: Bootstrap starting.
I20260620 23:35:44.875861 2700 tablet_bootstrap.cc:492] T 3c699dc2009040098a7e455f738f84b6 P f10804aa6b1d4b22a5a82fb8e2e9d0d6: Bootstrap starting.
I20260620 23:35:44.876508 2695 ts_tablet_manager.cc:1905] T 2cabd8969ea749c0a532055bef9877ef P 1fcac2741ad84a55bc68e5dd99ae3bd5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:35:44.877712 2699 tablet_bootstrap.cc:654] T 3c699dc2009040098a7e455f738f84b6 P fa4063ffe74c4ef183d404cf1d2c20cd: Neither blocks nor log segments found. Creating new log.
I20260620 23:35:44.882001 2698 tablet_bootstrap.cc:492] T 3c699dc2009040098a7e455f738f84b6 P 1fcac2741ad84a55bc68e5dd99ae3bd5: Bootstrap starting.
I20260620 23:35:44.882196 2696 tablet_replica.cc:330] T 2cabd8969ea749c0a532055bef9877ef P fa4063ffe74c4ef183d404cf1d2c20cd: stopping tablet replica
I20260620 23:35:44.882658 2696 raft_consensus.cc:2229] T 2cabd8969ea749c0a532055bef9877ef P fa4063ffe74c4ef183d404cf1d2c20cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:35:44.883066 2696 raft_consensus.cc:2258] T 2cabd8969ea749c0a532055bef9877ef P fa4063ffe74c4ef183d404cf1d2c20cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:35:44.883124 2700 tablet_bootstrap.cc:654] T 3c699dc2009040098a7e455f738f84b6 P f10804aa6b1d4b22a5a82fb8e2e9d0d6: Neither blocks nor log segments found. Creating new log.
I20260620 23:35:44.883898 2697 tablet_replica.cc:330] T 2cabd8969ea749c0a532055bef9877ef P f10804aa6b1d4b22a5a82fb8e2e9d0d6: stopping tablet replica
I20260620 23:35:44.884372 2697 raft_consensus.cc:2229] T 2cabd8969ea749c0a532055bef9877ef P f10804aa6b1d4b22a5a82fb8e2e9d0d6 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:35:44.884851 2698 tablet_bootstrap.cc:654] T 3c699dc2009040098a7e455f738f84b6 P 1fcac2741ad84a55bc68e5dd99ae3bd5: Neither blocks nor log segments found. Creating new log.
I20260620 23:35:44.885074 2697 raft_consensus.cc:2258] T 2cabd8969ea749c0a532055bef9877ef P f10804aa6b1d4b22a5a82fb8e2e9d0d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:35:44.887955 2698 tablet_bootstrap.cc:492] T 3c699dc2009040098a7e455f738f84b6 P 1fcac2741ad84a55bc68e5dd99ae3bd5: No bootstrap required, opened a new log
I20260620 23:35:44.888182 2698 ts_tablet_manager.cc:1397] T 3c699dc2009040098a7e455f738f84b6 P 1fcac2741ad84a55bc68e5dd99ae3bd5: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 23:35:44.889040 2698 raft_consensus.cc:348] T 3c699dc2009040098a7e455f738f84b6 P 1fcac2741ad84a55bc68e5dd99ae3bd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40819 } } peers { permanent_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33641 } } peers { permanent_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37077 } }
I20260620 23:35:44.889474 2698 raft_consensus.cc:374] T 3c699dc2009040098a7e455f738f84b6 P 1fcac2741ad84a55bc68e5dd99ae3bd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:35:44.889609 2698 raft_consensus.cc:729] T 3c699dc2009040098a7e455f738f84b6 P 1fcac2741ad84a55bc68e5dd99ae3bd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fcac2741ad84a55bc68e5dd99ae3bd5, State: Initialized, Role: FOLLOWER
I20260620 23:35:44.889955 2698 consensus_queue.cc:260] T 3c699dc2009040098a7e455f738f84b6 P 1fcac2741ad84a55bc68e5dd99ae3bd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40819 } } peers { permanent_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33641 } } peers { permanent_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37077 } }
I20260620 23:35:44.891814 2699 tablet_bootstrap.cc:492] T 3c699dc2009040098a7e455f738f84b6 P fa4063ffe74c4ef183d404cf1d2c20cd: No bootstrap required, opened a new log
I20260620 23:35:44.891964 2699 ts_tablet_manager.cc:1397] T 3c699dc2009040098a7e455f738f84b6 P fa4063ffe74c4ef183d404cf1d2c20cd: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20260620 23:35:44.892650 2697 ts_tablet_manager.cc:1905] T 2cabd8969ea749c0a532055bef9877ef P f10804aa6b1d4b22a5a82fb8e2e9d0d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:35:44.892803 2699 raft_consensus.cc:348] T 3c699dc2009040098a7e455f738f84b6 P fa4063ffe74c4ef183d404cf1d2c20cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40819 } } peers { permanent_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33641 } } peers { permanent_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37077 } }
I20260620 23:35:44.893119 2699 raft_consensus.cc:374] T 3c699dc2009040098a7e455f738f84b6 P fa4063ffe74c4ef183d404cf1d2c20cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:35:44.893211 2699 raft_consensus.cc:729] T 3c699dc2009040098a7e455f738f84b6 P fa4063ffe74c4ef183d404cf1d2c20cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa4063ffe74c4ef183d404cf1d2c20cd, State: Initialized, Role: FOLLOWER
I20260620 23:35:44.893468 2699 consensus_queue.cc:260] T 3c699dc2009040098a7e455f738f84b6 P fa4063ffe74c4ef183d404cf1d2c20cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40819 } } peers { permanent_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33641 } } peers { permanent_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37077 } }
I20260620 23:35:44.893663 2698 ts_tablet_manager.cc:1428] T 3c699dc2009040098a7e455f738f84b6 P 1fcac2741ad84a55bc68e5dd99ae3bd5: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260620 23:35:44.894183 2699 ts_tablet_manager.cc:1428] T 3c699dc2009040098a7e455f738f84b6 P fa4063ffe74c4ef183d404cf1d2c20cd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:35:44.899580 2696 ts_tablet_manager.cc:1905] T 2cabd8969ea749c0a532055bef9877ef P fa4063ffe74c4ef183d404cf1d2c20cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:35:44.905277 2700 tablet_bootstrap.cc:492] T 3c699dc2009040098a7e455f738f84b6 P f10804aa6b1d4b22a5a82fb8e2e9d0d6: No bootstrap required, opened a new log
I20260620 23:35:44.905504 2700 ts_tablet_manager.cc:1397] T 3c699dc2009040098a7e455f738f84b6 P f10804aa6b1d4b22a5a82fb8e2e9d0d6: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.000s
I20260620 23:35:44.907734 2700 raft_consensus.cc:348] T 3c699dc2009040098a7e455f738f84b6 P f10804aa6b1d4b22a5a82fb8e2e9d0d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40819 } } peers { permanent_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33641 } } peers { permanent_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37077 } }
I20260620 23:35:44.908048 2700 raft_consensus.cc:374] T 3c699dc2009040098a7e455f738f84b6 P f10804aa6b1d4b22a5a82fb8e2e9d0d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:35:44.908166 2700 raft_consensus.cc:729] T 3c699dc2009040098a7e455f738f84b6 P f10804aa6b1d4b22a5a82fb8e2e9d0d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f10804aa6b1d4b22a5a82fb8e2e9d0d6, State: Initialized, Role: FOLLOWER
I20260620 23:35:44.908474 2700 consensus_queue.cc:260] T 3c699dc2009040098a7e455f738f84b6 P f10804aa6b1d4b22a5a82fb8e2e9d0d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40819 } } peers { permanent_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33641 } } peers { permanent_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37077 } }
I20260620 23:35:44.911556 2700 ts_tablet_manager.cc:1428] T 3c699dc2009040098a7e455f738f84b6 P f10804aa6b1d4b22a5a82fb8e2e9d0d6: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260620 23:35:44.939168 2695 ts_tablet_manager.cc:1918] T 2cabd8969ea749c0a532055bef9877ef P 1fcac2741ad84a55bc68e5dd99ae3bd5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:35:44.939188 2696 ts_tablet_manager.cc:1918] T 2cabd8969ea749c0a532055bef9877ef P fa4063ffe74c4ef183d404cf1d2c20cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:35:44.939378 2696 log.cc:1196] T 2cabd8969ea749c0a532055bef9877ef P fa4063ffe74c4ef183d404cf1d2c20cd: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/ts-2/wal/wals/2cabd8969ea749c0a532055bef9877ef
I20260620 23:35:44.939574 2695 log.cc:1196] T 2cabd8969ea749c0a532055bef9877ef P 1fcac2741ad84a55bc68e5dd99ae3bd5: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/ts-0/wal/wals/2cabd8969ea749c0a532055bef9877ef
I20260620 23:35:44.939941 2696 ts_tablet_manager.cc:1939] T 2cabd8969ea749c0a532055bef9877ef P fa4063ffe74c4ef183d404cf1d2c20cd: Deleting consensus metadata
I20260620 23:35:44.940272 2695 ts_tablet_manager.cc:1939] T 2cabd8969ea749c0a532055bef9877ef P 1fcac2741ad84a55bc68e5dd99ae3bd5: Deleting consensus metadata
I20260620 23:35:44.954883 2649 raft_consensus.cc:482] T 3c699dc2009040098a7e455f738f84b6 P fa4063ffe74c4ef183d404cf1d2c20cd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:35:44.955188 2649 raft_consensus.cc:504] T 3c699dc2009040098a7e455f738f84b6 P fa4063ffe74c4ef183d404cf1d2c20cd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40819 } } peers { permanent_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33641 } } peers { permanent_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37077 } }
I20260620 23:35:44.957096 2649 leader_election.cc:288] T 3c699dc2009040098a7e455f738f84b6 P fa4063ffe74c4ef183d404cf1d2c20cd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1fcac2741ad84a55bc68e5dd99ae3bd5 (127.0.97.1:40819), f10804aa6b1d4b22a5a82fb8e2e9d0d6 (127.0.97.2:33641)
I20260620 23:35:44.977134 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:44.957696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49061 (local address 127.0.97.1:40819)
0620 23:35:44.957895 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:44.957902 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:44.968960 (+ 11058us) server_negotiation.cc:407] Connection header received
0620 23:35:44.969465 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:44.969472 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:44.969563 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:44.969637 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:35:44.970387 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:44.971620 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:44.974337 (+ 2717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:44.975822 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:44.975927 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:44.976047 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:44.976612 (+ 565us) server_negotiation.cc:299] Negotiation successful
0620 23:35:44.976773 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 23:35:44.978130 2705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:44.957322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40819 (local address 127.0.97.3:49061)
0620 23:35:44.968828 (+ 11506us) negotiation.cc:107] Waiting for socket to connect
0620 23:35:44.968853 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 23:35:44.969281 (+ 428us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:35:44.969833 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:35:44.969838 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:35:44.969863 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:35:44.970204 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:44.970216 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:44.971819 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:44.971824 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:44.974160 (+ 2336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:44.974175 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:44.976222 (+ 2047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:44.976228 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:44.976416 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:44.976450 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 23:35:44.977638 (+ 1188us) client_negotiation.cc:240] Negotiation successful
0620 23:35:44.977798 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11351,"thread_start_us":14633,"threads_started":1}
I20260620 23:35:44.979105 2296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c699dc2009040098a7e455f738f84b6" candidate_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" is_pre_election: true
I20260620 23:35:44.979388 2296 raft_consensus.cc:2454] T 3c699dc2009040098a7e455f738f84b6 P 1fcac2741ad84a55bc68e5dd99ae3bd5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa4063ffe74c4ef183d404cf1d2c20cd in term 0.
I20260620 23:35:44.979933 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:44.957995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45725 (local address 127.0.97.2:33641)
0620 23:35:44.958227 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:44.958235 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:44.971888 (+ 13653us) server_negotiation.cc:407] Connection header received
0620 23:35:44.972536 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:44.972544 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:44.972634 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:44.972709 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:35:44.973749 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:44.974879 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:44.977204 (+ 2325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:44.978579 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:44.978708 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:44.978813 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:44.979503 (+ 690us) server_negotiation.cc:299] Negotiation successful
0620 23:35:44.979652 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 23:35:44.980207 2505 leader_election.cc:302] T 3c699dc2009040098a7e455f738f84b6 P fa4063ffe74c4ef183d404cf1d2c20cd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fcac2741ad84a55bc68e5dd99ae3bd5, fa4063ffe74c4ef183d404cf1d2c20cd; no voters:
I20260620 23:35:44.980693 2649 raft_consensus.cc:2781] T 3c699dc2009040098a7e455f738f84b6 P fa4063ffe74c4ef183d404cf1d2c20cd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:35:44.980754 2704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:44.957654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33641 (local address 127.0.97.3:45725)
0620 23:35:44.971769 (+ 14115us) negotiation.cc:107] Waiting for socket to connect
0620 23:35:44.971792 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 23:35:44.972288 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:35:44.972902 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:35:44.972907 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:35:44.972932 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:35:44.973547 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:44.973563 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:44.975079 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:44.975084 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:44.977025 (+ 1941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:44.977038 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:44.979048 (+ 2010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:44.979054 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:44.979293 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:44.979331 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 23:35:44.980233 (+ 902us) client_negotiation.cc:240] Negotiation successful
0620 23:35:44.980400 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13959,"thread_start_us":11331,"threads_started":1}
I20260620 23:35:44.980816 2649 raft_consensus.cc:482] T 3c699dc2009040098a7e455f738f84b6 P fa4063ffe74c4ef183d404cf1d2c20cd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:35:44.980911 2649 raft_consensus.cc:3037] T 3c699dc2009040098a7e455f738f84b6 P fa4063ffe74c4ef183d404cf1d2c20cd [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:35:44.981535 2435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c699dc2009040098a7e455f738f84b6" candidate_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" is_pre_election: true
I20260620 23:35:44.981803 2435 raft_consensus.cc:2454] T 3c699dc2009040098a7e455f738f84b6 P f10804aa6b1d4b22a5a82fb8e2e9d0d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa4063ffe74c4ef183d404cf1d2c20cd in term 0.
I20260620 23:35:45.021647 2123 catalog_manager.cc:4841] TS 1fcac2741ad84a55bc68e5dd99ae3bd5 (127.0.97.1:40819): tablet 2cabd8969ea749c0a532055bef9877ef (table test-table [id=1fa3bc81b1954cf5b78be7d7c3091b1d]) successfully deleted
I20260620 23:35:45.022166 2697 ts_tablet_manager.cc:1918] T 2cabd8969ea749c0a532055bef9877ef P f10804aa6b1d4b22a5a82fb8e2e9d0d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:35:45.022377 2697 log.cc:1196] T 2cabd8969ea749c0a532055bef9877ef P f10804aa6b1d4b22a5a82fb8e2e9d0d6: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/ts-1/wal/wals/2cabd8969ea749c0a532055bef9877ef
I20260620 23:35:45.023025 2697 ts_tablet_manager.cc:1939] T 2cabd8969ea749c0a532055bef9877ef P f10804aa6b1d4b22a5a82fb8e2e9d0d6: Deleting consensus metadata
I20260620 23:35:45.023932 2649 raft_consensus.cc:504] T 3c699dc2009040098a7e455f738f84b6 P fa4063ffe74c4ef183d404cf1d2c20cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40819 } } peers { permanent_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33641 } } peers { permanent_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37077 } }
I20260620 23:35:45.024745 2126 catalog_manager.cc:4841] TS f10804aa6b1d4b22a5a82fb8e2e9d0d6 (127.0.97.2:33641): tablet 2cabd8969ea749c0a532055bef9877ef (table test-table [id=1fa3bc81b1954cf5b78be7d7c3091b1d]) successfully deleted
I20260620 23:35:45.025074 2123 catalog_manager.cc:4841] TS fa4063ffe74c4ef183d404cf1d2c20cd (127.0.97.3:37077): tablet 2cabd8969ea749c0a532055bef9877ef (table test-table [id=1fa3bc81b1954cf5b78be7d7c3091b1d]) successfully deleted
I20260620 23:35:45.025420 2649 leader_election.cc:288] T 3c699dc2009040098a7e455f738f84b6 P fa4063ffe74c4ef183d404cf1d2c20cd [CANDIDATE]: Term 1 election: Requested vote from peers 1fcac2741ad84a55bc68e5dd99ae3bd5 (127.0.97.1:40819), f10804aa6b1d4b22a5a82fb8e2e9d0d6 (127.0.97.2:33641)
I20260620 23:35:45.025455 2296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c699dc2009040098a7e455f738f84b6" candidate_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5"
I20260620 23:35:45.025722 2296 raft_consensus.cc:3037] T 3c699dc2009040098a7e455f738f84b6 P 1fcac2741ad84a55bc68e5dd99ae3bd5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:35:45.026265 2435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c699dc2009040098a7e455f738f84b6" candidate_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6"
I20260620 23:35:45.026508 2435 raft_consensus.cc:3037] T 3c699dc2009040098a7e455f738f84b6 P f10804aa6b1d4b22a5a82fb8e2e9d0d6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:35:45.029152 2296 raft_consensus.cc:2454] T 3c699dc2009040098a7e455f738f84b6 P 1fcac2741ad84a55bc68e5dd99ae3bd5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa4063ffe74c4ef183d404cf1d2c20cd in term 1.
I20260620 23:35:45.029675 2435 raft_consensus.cc:2454] T 3c699dc2009040098a7e455f738f84b6 P f10804aa6b1d4b22a5a82fb8e2e9d0d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa4063ffe74c4ef183d404cf1d2c20cd in term 1.
I20260620 23:35:45.029754 2505 leader_election.cc:302] T 3c699dc2009040098a7e455f738f84b6 P fa4063ffe74c4ef183d404cf1d2c20cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fcac2741ad84a55bc68e5dd99ae3bd5, fa4063ffe74c4ef183d404cf1d2c20cd; no voters:
I20260620 23:35:45.030128 2649 raft_consensus.cc:2781] T 3c699dc2009040098a7e455f738f84b6 P fa4063ffe74c4ef183d404cf1d2c20cd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:35:45.030651 2649 raft_consensus.cc:686] T 3c699dc2009040098a7e455f738f84b6 P fa4063ffe74c4ef183d404cf1d2c20cd [term 1 LEADER]: Becoming Leader. State: Replica: fa4063ffe74c4ef183d404cf1d2c20cd, State: Running, Role: LEADER
I20260620 23:35:45.031172 2649 consensus_queue.cc:237] T 3c699dc2009040098a7e455f738f84b6 P fa4063ffe74c4ef183d404cf1d2c20cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40819 } } peers { permanent_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33641 } } peers { permanent_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37077 } }
I20260620 23:35:45.035269 2138 catalog_manager.cc:5495] T 3c699dc2009040098a7e455f738f84b6 P fa4063ffe74c4ef183d404cf1d2c20cd reported cstate change: term changed from 0 to 1, leader changed from <none> to fa4063ffe74c4ef183d404cf1d2c20cd (127.0.97.3). New cstate: current_term: 1 leader_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33641 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37077 } health_report { overall_health: HEALTHY } } }
I20260620 23:35:45.090515 2708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:45.077324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44514 (local address 127.0.97.3:37077)
0620 23:35:45.083449 (+ 6125us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:45.083456 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:45.083486 (+ 30us) server_negotiation.cc:407] Connection header received
0620 23:35:45.083598 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:45.083604 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:45.083726 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:45.083927 (+ 201us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:35:45.084424 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:45.085578 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:45.087589 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:45.088229 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:45.088361 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:45.088887 (+ 526us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:35:45.089759 (+ 872us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:35:45.089917 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:45.090048 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 23:35:45.090132 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5957,"thread_start_us":5817,"threads_started":1}
I20260620 23:35:45.114867 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:45.103134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34575 (local address 127.0.97.1:40819)
0620 23:35:45.103318 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:45.103325 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:45.107727 (+ 4402us) server_negotiation.cc:407] Connection header received
0620 23:35:45.107948 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:45.107954 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:45.108048 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:45.108160 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:35:45.109223 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:45.110407 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:45.112543 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:45.113275 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:45.113406 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:45.114001 (+ 595us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:35:45.114158 (+ 157us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:35:45.114247 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:45.114465 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 23:35:45.114565 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 23:35:45.115468 2711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:45.102921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40819 (local address 127.0.97.3:34575)
0620 23:35:45.107571 (+ 4650us) negotiation.cc:107] Waiting for socket to connect
0620 23:35:45.107596 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 23:35:45.107797 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:35:45.108191 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:35:45.108197 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:35:45.108209 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:35:45.109022 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:45.109040 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:45.110580 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:45.110586 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:45.112349 (+ 1763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:45.112368 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:45.113496 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:45.113503 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:45.113696 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:45.113792 (+ 96us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:35:45.114276 (+ 484us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:35:45.114289 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 23:35:45.114429 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 23:35:45.114587 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4453,"thread_start_us":4233,"threads_started":1}
I20260620 23:35:45.127106 2435 raft_consensus.cc:1261] T 3c699dc2009040098a7e455f738f84b6 P f10804aa6b1d4b22a5a82fb8e2e9d0d6 [term 1 FOLLOWER]: Refusing update from remote peer fa4063ffe74c4ef183d404cf1d2c20cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:35:45.127640 2296 raft_consensus.cc:1261] T 3c699dc2009040098a7e455f738f84b6 P 1fcac2741ad84a55bc68e5dd99ae3bd5 [term 1 FOLLOWER]: Refusing update from remote peer fa4063ffe74c4ef183d404cf1d2c20cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:35:45.127920 2706 consensus_queue.cc:1035] T 3c699dc2009040098a7e455f738f84b6 P fa4063ffe74c4ef183d404cf1d2c20cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:35:45.128357 2649 consensus_queue.cc:1035] T 3c699dc2009040098a7e455f738f84b6 P fa4063ffe74c4ef183d404cf1d2c20cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:35:45.142789 2677 mvcc.cc:205] Tried to move back new op lower bound from 7299066040834146304 to 7299066040452308992. Current Snapshot: MvccSnapshot[applied={T|T < 7299066040834146304}]
I20260620 23:35:45.171185 2708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:45.164621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52017 (local address 127.0.97.3:37077)
0620 23:35:45.164878 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:45.164884 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:45.164899 (+ 15us) server_negotiation.cc:407] Connection header received
0620 23:35:45.164955 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:45.164960 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:45.165035 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:45.165144 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:35:45.165573 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:45.166528 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:45.169034 (+ 2506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:45.169762 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:45.169894 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:45.170422 (+ 528us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:35:45.170561 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:35:45.170653 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:45.170815 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 23:35:45.170879 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148}
I20260620 23:35:45.171439 2691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:45.164258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37077 (local address 127.0.97.1:52017)
0620 23:35:45.164523 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0620 23:35:45.164543 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:35:45.164657 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:35:45.165162 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:35:45.165167 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:35:45.165180 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:35:45.165399 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:45.165410 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:45.166679 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:45.166685 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:45.168835 (+ 2150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:45.168853 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:45.170013 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:45.170019 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:45.170205 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:45.170262 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:35:45.170686 (+ 424us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:35:45.170699 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 23:35:45.170799 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 23:35:45.170985 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115}
W20260620 23:35:45.172547 2677 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 23:35:46.808794 2138 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37178:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:35:46.809046 2138 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37178:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:35:46.813019 2138 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 78b3f3598f0540858739235691fbe934: Sending DeleteTablet for 3 replicas of tablet 3c699dc2009040098a7e455f738f84b6
I20260620 23:35:46.813918 2275 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c699dc2009040098a7e455f738f84b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:35:46 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47690
I20260620 23:35:46.813952 2554 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c699dc2009040098a7e455f738f84b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:35:46 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44508
I20260620 23:35:46.813941 2415 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c699dc2009040098a7e455f738f84b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:35:46 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37478
I20260620 23:35:46.825173 2742 tablet_replica.cc:330] T 3c699dc2009040098a7e455f738f84b6 P f10804aa6b1d4b22a5a82fb8e2e9d0d6: stopping tablet replica
I20260620 23:35:46.825588 2742 raft_consensus.cc:2229] T 3c699dc2009040098a7e455f738f84b6 P f10804aa6b1d4b22a5a82fb8e2e9d0d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:35:46.825951 2742 raft_consensus.cc:2258] T 3c699dc2009040098a7e455f738f84b6 P f10804aa6b1d4b22a5a82fb8e2e9d0d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:35:46.826129 2740 tablet_replica.cc:330] T 3c699dc2009040098a7e455f738f84b6 P 1fcac2741ad84a55bc68e5dd99ae3bd5: stopping tablet replica
I20260620 23:35:46.826349 2740 raft_consensus.cc:2229] T 3c699dc2009040098a7e455f738f84b6 P 1fcac2741ad84a55bc68e5dd99ae3bd5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:35:46.826596 2740 raft_consensus.cc:2258] T 3c699dc2009040098a7e455f738f84b6 P 1fcac2741ad84a55bc68e5dd99ae3bd5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:35:46.827271 2742 ts_tablet_manager.cc:1905] T 3c699dc2009040098a7e455f738f84b6 P f10804aa6b1d4b22a5a82fb8e2e9d0d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:35:46.827656 2740 ts_tablet_manager.cc:1905] T 3c699dc2009040098a7e455f738f84b6 P 1fcac2741ad84a55bc68e5dd99ae3bd5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:35:46.828148 2741 tablet_replica.cc:330] T 3c699dc2009040098a7e455f738f84b6 P fa4063ffe74c4ef183d404cf1d2c20cd: stopping tablet replica
I20260620 23:35:46.828387 2741 raft_consensus.cc:2229] T 3c699dc2009040098a7e455f738f84b6 P fa4063ffe74c4ef183d404cf1d2c20cd [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:35:46.829036 2741 raft_consensus.cc:2258] T 3c699dc2009040098a7e455f738f84b6 P fa4063ffe74c4ef183d404cf1d2c20cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:35:46.829998 2741 ts_tablet_manager.cc:1905] T 3c699dc2009040098a7e455f738f84b6 P fa4063ffe74c4ef183d404cf1d2c20cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:35:46.834926 2740 ts_tablet_manager.cc:1918] T 3c699dc2009040098a7e455f738f84b6 P 1fcac2741ad84a55bc68e5dd99ae3bd5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 23:35:46.835161 2740 log.cc:1196] T 3c699dc2009040098a7e455f738f84b6 P 1fcac2741ad84a55bc68e5dd99ae3bd5: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/ts-0/wal/wals/3c699dc2009040098a7e455f738f84b6
I20260620 23:35:46.835655 2740 ts_tablet_manager.cc:1939] T 3c699dc2009040098a7e455f738f84b6 P 1fcac2741ad84a55bc68e5dd99ae3bd5: Deleting consensus metadata
I20260620 23:35:46.837498 2123 catalog_manager.cc:4841] TS 1fcac2741ad84a55bc68e5dd99ae3bd5 (127.0.97.1:40819): tablet 3c699dc2009040098a7e455f738f84b6 (table test-table [id=802cd0100ebf48f097ac82e39da5a451]) successfully deleted
I20260620 23:35:46.837551 2741 ts_tablet_manager.cc:1918] T 3c699dc2009040098a7e455f738f84b6 P fa4063ffe74c4ef183d404cf1d2c20cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 23:35:46.837697 2741 log.cc:1196] T 3c699dc2009040098a7e455f738f84b6 P fa4063ffe74c4ef183d404cf1d2c20cd: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/ts-2/wal/wals/3c699dc2009040098a7e455f738f84b6
I20260620 23:35:46.838171 2741 ts_tablet_manager.cc:1939] T 3c699dc2009040098a7e455f738f84b6 P fa4063ffe74c4ef183d404cf1d2c20cd: Deleting consensus metadata
I20260620 23:35:46.839426 2123 catalog_manager.cc:4841] TS fa4063ffe74c4ef183d404cf1d2c20cd (127.0.97.3:37077): tablet 3c699dc2009040098a7e455f738f84b6 (table test-table [id=802cd0100ebf48f097ac82e39da5a451]) successfully deleted
I20260620 23:35:46.840094 2742 ts_tablet_manager.cc:1918] T 3c699dc2009040098a7e455f738f84b6 P f10804aa6b1d4b22a5a82fb8e2e9d0d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 23:35:46.840230 2742 log.cc:1196] T 3c699dc2009040098a7e455f738f84b6 P f10804aa6b1d4b22a5a82fb8e2e9d0d6: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/ts-1/wal/wals/3c699dc2009040098a7e455f738f84b6
I20260620 23:35:46.840636 2742 ts_tablet_manager.cc:1939] T 3c699dc2009040098a7e455f738f84b6 P f10804aa6b1d4b22a5a82fb8e2e9d0d6: Deleting consensus metadata
I20260620 23:35:46.841701 2126 catalog_manager.cc:4841] TS f10804aa6b1d4b22a5a82fb8e2e9d0d6 (127.0.97.2:33641): tablet 3c699dc2009040098a7e455f738f84b6 (table test-table [id=802cd0100ebf48f097ac82e39da5a451]) successfully deleted
Jun 20 23:35:46 dist-test-slave-tcgx krb5kdc[2094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998546, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 23:35:46 dist-test-slave-tcgx krb5kdc[2094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998546, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:35:46.880278 2750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:46.865206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50072 (local address 127.0.97.62:43025)
0620 23:35:46.865801 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:46.865806 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:46.865828 (+ 22us) server_negotiation.cc:407] Connection header received
0620 23:35:46.865975 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:46.865979 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:46.866041 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:46.866136 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:35:46.867499 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:46.868298 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:46.869781 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:46.870221 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:46.870323 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:46.874663 (+ 4340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:35:46.874669 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:35:46.874673 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:35:46.874710 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:35:46.877731 (+ 3021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:46.878338 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:46.878342 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:46.878344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:46.878457 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:46.878798 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:46.878801 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:46.878803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:46.879139 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:35:46.879311 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:46.879636 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 23:35:46.879878 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":331,"threads_started":1}
I20260620 23:35:46.883925 2138 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50072:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:35:46.884470 2138 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:35:46.891160 2415 tablet_service.cc:1467] Processing CreateTablet for tablet 38d27dc611d64fc4bb42ce26cac089fa (DEFAULT_TABLE table=test-table [id=afe2ff961b6e4480b9a2614390fcf155]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:35:46.891111 2275 tablet_service.cc:1467] Processing CreateTablet for tablet 38d27dc611d64fc4bb42ce26cac089fa (DEFAULT_TABLE table=test-table [id=afe2ff961b6e4480b9a2614390fcf155]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:35:46.891566 2415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38d27dc611d64fc4bb42ce26cac089fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:35:46.891599 2275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38d27dc611d64fc4bb42ce26cac089fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:35:46.891484 2554 tablet_service.cc:1467] Processing CreateTablet for tablet 38d27dc611d64fc4bb42ce26cac089fa (DEFAULT_TABLE table=test-table [id=afe2ff961b6e4480b9a2614390fcf155]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:35:46.891950 2554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38d27dc611d64fc4bb42ce26cac089fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:35:46.902135 2751 tablet_bootstrap.cc:492] T 38d27dc611d64fc4bb42ce26cac089fa P f10804aa6b1d4b22a5a82fb8e2e9d0d6: Bootstrap starting.
I20260620 23:35:46.903623 2753 tablet_bootstrap.cc:492] T 38d27dc611d64fc4bb42ce26cac089fa P fa4063ffe74c4ef183d404cf1d2c20cd: Bootstrap starting.
I20260620 23:35:46.904078 2751 tablet_bootstrap.cc:654] T 38d27dc611d64fc4bb42ce26cac089fa P f10804aa6b1d4b22a5a82fb8e2e9d0d6: Neither blocks nor log segments found. Creating new log.
I20260620 23:35:46.908027 2753 tablet_bootstrap.cc:654] T 38d27dc611d64fc4bb42ce26cac089fa P fa4063ffe74c4ef183d404cf1d2c20cd: Neither blocks nor log segments found. Creating new log.
I20260620 23:35:46.911216 2752 tablet_bootstrap.cc:492] T 38d27dc611d64fc4bb42ce26cac089fa P 1fcac2741ad84a55bc68e5dd99ae3bd5: Bootstrap starting.
I20260620 23:35:46.913125 2751 tablet_bootstrap.cc:492] T 38d27dc611d64fc4bb42ce26cac089fa P f10804aa6b1d4b22a5a82fb8e2e9d0d6: No bootstrap required, opened a new log
I20260620 23:35:46.913344 2751 ts_tablet_manager.cc:1397] T 38d27dc611d64fc4bb42ce26cac089fa P f10804aa6b1d4b22a5a82fb8e2e9d0d6: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.007s
I20260620 23:35:46.914695 2751 raft_consensus.cc:348] T 38d27dc611d64fc4bb42ce26cac089fa P f10804aa6b1d4b22a5a82fb8e2e9d0d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40819 } } peers { permanent_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33641 } } peers { permanent_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37077 } }
I20260620 23:35:46.915127 2751 raft_consensus.cc:374] T 38d27dc611d64fc4bb42ce26cac089fa P f10804aa6b1d4b22a5a82fb8e2e9d0d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:35:46.915264 2751 raft_consensus.cc:729] T 38d27dc611d64fc4bb42ce26cac089fa P f10804aa6b1d4b22a5a82fb8e2e9d0d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f10804aa6b1d4b22a5a82fb8e2e9d0d6, State: Initialized, Role: FOLLOWER
I20260620 23:35:46.915268 2752 tablet_bootstrap.cc:654] T 38d27dc611d64fc4bb42ce26cac089fa P 1fcac2741ad84a55bc68e5dd99ae3bd5: Neither blocks nor log segments found. Creating new log.
I20260620 23:35:46.915606 2751 consensus_queue.cc:260] T 38d27dc611d64fc4bb42ce26cac089fa P f10804aa6b1d4b22a5a82fb8e2e9d0d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40819 } } peers { permanent_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33641 } } peers { permanent_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37077 } }
I20260620 23:35:46.920300 2751 ts_tablet_manager.cc:1428] T 38d27dc611d64fc4bb42ce26cac089fa P f10804aa6b1d4b22a5a82fb8e2e9d0d6: Time spent starting tablet: real 0.007s user 0.000s sys 0.006s
I20260620 23:35:46.920650 2753 tablet_bootstrap.cc:492] T 38d27dc611d64fc4bb42ce26cac089fa P fa4063ffe74c4ef183d404cf1d2c20cd: No bootstrap required, opened a new log
I20260620 23:35:46.920884 2753 ts_tablet_manager.cc:1397] T 38d27dc611d64fc4bb42ce26cac089fa P fa4063ffe74c4ef183d404cf1d2c20cd: Time spent bootstrapping tablet: real 0.017s user 0.013s sys 0.000s
I20260620 23:35:46.922019 2752 tablet_bootstrap.cc:492] T 38d27dc611d64fc4bb42ce26cac089fa P 1fcac2741ad84a55bc68e5dd99ae3bd5: No bootstrap required, opened a new log
I20260620 23:35:46.921885 2753 raft_consensus.cc:348] T 38d27dc611d64fc4bb42ce26cac089fa P fa4063ffe74c4ef183d404cf1d2c20cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40819 } } peers { permanent_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33641 } } peers { permanent_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37077 } }
I20260620 23:35:46.922243 2752 ts_tablet_manager.cc:1397] T 38d27dc611d64fc4bb42ce26cac089fa P 1fcac2741ad84a55bc68e5dd99ae3bd5: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260620 23:35:46.922245 2753 raft_consensus.cc:374] T 38d27dc611d64fc4bb42ce26cac089fa P fa4063ffe74c4ef183d404cf1d2c20cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:35:46.922396 2753 raft_consensus.cc:729] T 38d27dc611d64fc4bb42ce26cac089fa P fa4063ffe74c4ef183d404cf1d2c20cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa4063ffe74c4ef183d404cf1d2c20cd, State: Initialized, Role: FOLLOWER
I20260620 23:35:46.922783 2753 consensus_queue.cc:260] T 38d27dc611d64fc4bb42ce26cac089fa P fa4063ffe74c4ef183d404cf1d2c20cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40819 } } peers { permanent_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33641 } } peers { permanent_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37077 } }
I20260620 23:35:46.923203 2752 raft_consensus.cc:348] T 38d27dc611d64fc4bb42ce26cac089fa P 1fcac2741ad84a55bc68e5dd99ae3bd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40819 } } peers { permanent_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33641 } } peers { permanent_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37077 } }
I20260620 23:35:46.923493 2752 raft_consensus.cc:374] T 38d27dc611d64fc4bb42ce26cac089fa P 1fcac2741ad84a55bc68e5dd99ae3bd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:35:46.923568 2753 ts_tablet_manager.cc:1428] T 38d27dc611d64fc4bb42ce26cac089fa P fa4063ffe74c4ef183d404cf1d2c20cd: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 23:35:46.923656 2752 raft_consensus.cc:729] T 38d27dc611d64fc4bb42ce26cac089fa P 1fcac2741ad84a55bc68e5dd99ae3bd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fcac2741ad84a55bc68e5dd99ae3bd5, State: Initialized, Role: FOLLOWER
I20260620 23:35:46.923983 2752 consensus_queue.cc:260] T 38d27dc611d64fc4bb42ce26cac089fa P 1fcac2741ad84a55bc68e5dd99ae3bd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40819 } } peers { permanent_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33641 } } peers { permanent_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37077 } }
I20260620 23:35:46.924629 2752 ts_tablet_manager.cc:1428] T 38d27dc611d64fc4bb42ce26cac089fa P 1fcac2741ad84a55bc68e5dd99ae3bd5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 23:35:47.002492 2718 raft_consensus.cc:482] T 38d27dc611d64fc4bb42ce26cac089fa P 1fcac2741ad84a55bc68e5dd99ae3bd5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:35:47.002674 2718 raft_consensus.cc:504] T 38d27dc611d64fc4bb42ce26cac089fa P 1fcac2741ad84a55bc68e5dd99ae3bd5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40819 } } peers { permanent_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33641 } } peers { permanent_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37077 } }
I20260620 23:35:47.003732 2718 leader_election.cc:288] T 38d27dc611d64fc4bb42ce26cac089fa P 1fcac2741ad84a55bc68e5dd99ae3bd5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f10804aa6b1d4b22a5a82fb8e2e9d0d6 (127.0.97.2:33641), fa4063ffe74c4ef183d404cf1d2c20cd (127.0.97.3:37077)
I20260620 23:35:47.004247 2434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38d27dc611d64fc4bb42ce26cac089fa" candidate_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" is_pre_election: true
I20260620 23:35:47.004274 2573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38d27dc611d64fc4bb42ce26cac089fa" candidate_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" is_pre_election: true
I20260620 23:35:47.004487 2434 raft_consensus.cc:2454] T 38d27dc611d64fc4bb42ce26cac089fa P f10804aa6b1d4b22a5a82fb8e2e9d0d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fcac2741ad84a55bc68e5dd99ae3bd5 in term 0.
I20260620 23:35:47.004535 2573 raft_consensus.cc:2454] T 38d27dc611d64fc4bb42ce26cac089fa P fa4063ffe74c4ef183d404cf1d2c20cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fcac2741ad84a55bc68e5dd99ae3bd5 in term 0.
I20260620 23:35:47.005005 2229 leader_election.cc:302] T 38d27dc611d64fc4bb42ce26cac089fa P 1fcac2741ad84a55bc68e5dd99ae3bd5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fcac2741ad84a55bc68e5dd99ae3bd5, f10804aa6b1d4b22a5a82fb8e2e9d0d6; no voters:
I20260620 23:35:47.005278 2718 raft_consensus.cc:2781] T 38d27dc611d64fc4bb42ce26cac089fa P 1fcac2741ad84a55bc68e5dd99ae3bd5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:35:47.005419 2718 raft_consensus.cc:482] T 38d27dc611d64fc4bb42ce26cac089fa P 1fcac2741ad84a55bc68e5dd99ae3bd5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:35:47.005513 2718 raft_consensus.cc:3037] T 38d27dc611d64fc4bb42ce26cac089fa P 1fcac2741ad84a55bc68e5dd99ae3bd5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:35:47.007620 2718 raft_consensus.cc:504] T 38d27dc611d64fc4bb42ce26cac089fa P 1fcac2741ad84a55bc68e5dd99ae3bd5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40819 } } peers { permanent_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33641 } } peers { permanent_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37077 } }
I20260620 23:35:47.008267 2718 leader_election.cc:288] T 38d27dc611d64fc4bb42ce26cac089fa P 1fcac2741ad84a55bc68e5dd99ae3bd5 [CANDIDATE]: Term 1 election: Requested vote from peers f10804aa6b1d4b22a5a82fb8e2e9d0d6 (127.0.97.2:33641), fa4063ffe74c4ef183d404cf1d2c20cd (127.0.97.3:37077)
I20260620 23:35:47.008731 2434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38d27dc611d64fc4bb42ce26cac089fa" candidate_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6"
I20260620 23:35:47.008803 2573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38d27dc611d64fc4bb42ce26cac089fa" candidate_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd"
I20260620 23:35:47.008955 2434 raft_consensus.cc:3037] T 38d27dc611d64fc4bb42ce26cac089fa P f10804aa6b1d4b22a5a82fb8e2e9d0d6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:35:47.009001 2573 raft_consensus.cc:3037] T 38d27dc611d64fc4bb42ce26cac089fa P fa4063ffe74c4ef183d404cf1d2c20cd [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:35:47.011967 2573 raft_consensus.cc:2454] T 38d27dc611d64fc4bb42ce26cac089fa P fa4063ffe74c4ef183d404cf1d2c20cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fcac2741ad84a55bc68e5dd99ae3bd5 in term 1.
I20260620 23:35:47.011977 2434 raft_consensus.cc:2454] T 38d27dc611d64fc4bb42ce26cac089fa P f10804aa6b1d4b22a5a82fb8e2e9d0d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fcac2741ad84a55bc68e5dd99ae3bd5 in term 1.
I20260620 23:35:47.012480 2226 leader_election.cc:302] T 38d27dc611d64fc4bb42ce26cac089fa P 1fcac2741ad84a55bc68e5dd99ae3bd5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fcac2741ad84a55bc68e5dd99ae3bd5, fa4063ffe74c4ef183d404cf1d2c20cd; no voters:
I20260620 23:35:47.012835 2718 raft_consensus.cc:2781] T 38d27dc611d64fc4bb42ce26cac089fa P 1fcac2741ad84a55bc68e5dd99ae3bd5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:35:47.013029 2718 raft_consensus.cc:686] T 38d27dc611d64fc4bb42ce26cac089fa P 1fcac2741ad84a55bc68e5dd99ae3bd5 [term 1 LEADER]: Becoming Leader. State: Replica: 1fcac2741ad84a55bc68e5dd99ae3bd5, State: Running, Role: LEADER
I20260620 23:35:47.013339 2718 consensus_queue.cc:237] T 38d27dc611d64fc4bb42ce26cac089fa P 1fcac2741ad84a55bc68e5dd99ae3bd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40819 } } peers { permanent_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33641 } } peers { permanent_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37077 } }
I20260620 23:35:47.016145 2138 catalog_manager.cc:5495] T 38d27dc611d64fc4bb42ce26cac089fa P 1fcac2741ad84a55bc68e5dd99ae3bd5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1fcac2741ad84a55bc68e5dd99ae3bd5 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40819 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33641 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37077 } health_report { overall_health: UNKNOWN } } }
I20260620 23:35:47.041919 2760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:47.036827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53434 (local address 127.0.97.1:40819)
0620 23:35:47.037352 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:47.037357 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:47.037376 (+ 19us) server_negotiation.cc:407] Connection header received
0620 23:35:47.037426 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:47.037429 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:47.037487 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:47.037570 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:35:47.037868 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:47.039075 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:47.040346 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:47.040761 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:47.040863 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:47.041277 (+ 414us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:35:47.041390 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:35:47.041469 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:47.041601 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 23:35:47.041668 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":298,"threads_started":1}
I20260620 23:35:47.055471 2760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:47.050478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56469 (local address 127.0.97.1:40819)
0620 23:35:47.050695 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:47.050701 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:47.050871 (+ 170us) server_negotiation.cc:407] Connection header received
0620 23:35:47.051113 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:47.051125 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:47.051177 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:47.051267 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:35:47.051677 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:47.052426 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:47.053819 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:47.054216 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:47.054368 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:47.054757 (+ 389us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:35:47.054848 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:35:47.054938 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:47.055091 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 23:35:47.055155 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 23:35:47.055534 2763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:47.050204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40819 (local address 127.0.97.1:56469)
0620 23:35:47.050782 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0620 23:35:47.050796 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:35:47.050990 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:35:47.051276 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:35:47.051280 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:35:47.051291 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:35:47.051551 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:47.051560 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:47.052569 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:47.052574 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:47.053673 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:47.053684 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:47.054403 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:47.054409 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:47.054573 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:47.054605 (+ 32us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:35:47.054967 (+ 362us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:35:47.054978 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 23:35:47.055075 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 23:35:47.055196 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":302,"threads_started":1}
I20260620 23:35:47.063166 2573 raft_consensus.cc:1261] T 38d27dc611d64fc4bb42ce26cac089fa P fa4063ffe74c4ef183d404cf1d2c20cd [term 1 FOLLOWER]: Refusing update from remote peer 1fcac2741ad84a55bc68e5dd99ae3bd5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:35:47.063251 2434 raft_consensus.cc:1261] T 38d27dc611d64fc4bb42ce26cac089fa P f10804aa6b1d4b22a5a82fb8e2e9d0d6 [term 1 FOLLOWER]: Refusing update from remote peer 1fcac2741ad84a55bc68e5dd99ae3bd5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:35:47.063654 2718 consensus_queue.cc:1035] T 38d27dc611d64fc4bb42ce26cac089fa P 1fcac2741ad84a55bc68e5dd99ae3bd5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:35:47.063902 2735 consensus_queue.cc:1035] T 38d27dc611d64fc4bb42ce26cac089fa P 1fcac2741ad84a55bc68e5dd99ae3bd5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:35:47.110744 2138 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:35:47.111059 2138 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:35:47.114297 2138 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 78b3f3598f0540858739235691fbe934: Sending DeleteTablet for 3 replicas of tablet 38d27dc611d64fc4bb42ce26cac089fa
I20260620 23:35:47.115285 2554 tablet_service.cc:1514] Processing DeleteTablet for tablet 38d27dc611d64fc4bb42ce26cac089fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:35:47 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44508
I20260620 23:35:47.115583 2741 tablet_replica.cc:330] T 38d27dc611d64fc4bb42ce26cac089fa P fa4063ffe74c4ef183d404cf1d2c20cd: stopping tablet replica
I20260620 23:35:47.115785 2741 raft_consensus.cc:2229] T 38d27dc611d64fc4bb42ce26cac089fa P fa4063ffe74c4ef183d404cf1d2c20cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:35:47.115835 2415 tablet_service.cc:1514] Processing DeleteTablet for tablet 38d27dc611d64fc4bb42ce26cac089fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:35:47 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37478
I20260620 23:35:47.115983 2741 raft_consensus.cc:2258] T 38d27dc611d64fc4bb42ce26cac089fa P fa4063ffe74c4ef183d404cf1d2c20cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:35:47.116248 2742 tablet_replica.cc:330] T 38d27dc611d64fc4bb42ce26cac089fa P f10804aa6b1d4b22a5a82fb8e2e9d0d6: stopping tablet replica
I20260620 23:35:47.116434 2742 raft_consensus.cc:2229] T 38d27dc611d64fc4bb42ce26cac089fa P f10804aa6b1d4b22a5a82fb8e2e9d0d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:35:47.116612 2742 raft_consensus.cc:2258] T 38d27dc611d64fc4bb42ce26cac089fa P f10804aa6b1d4b22a5a82fb8e2e9d0d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:35:47.116982 2138 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50072:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 23:35:47.117297 2741 ts_tablet_manager.cc:1905] T 38d27dc611d64fc4bb42ce26cac089fa P fa4063ffe74c4ef183d404cf1d2c20cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:35:47.117504 2276 tablet_service.cc:1514] Processing DeleteTablet for tablet 38d27dc611d64fc4bb42ce26cac089fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:35:47 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47690
I20260620 23:35:47.117784 2740 tablet_replica.cc:330] T 38d27dc611d64fc4bb42ce26cac089fa P 1fcac2741ad84a55bc68e5dd99ae3bd5: stopping tablet replica
W20260620 23:35:47.117945 2138 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:35:47.117980 2740 raft_consensus.cc:2229] T 38d27dc611d64fc4bb42ce26cac089fa P 1fcac2741ad84a55bc68e5dd99ae3bd5 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:35:47.118532 2740 raft_consensus.cc:2258] T 38d27dc611d64fc4bb42ce26cac089fa P 1fcac2741ad84a55bc68e5dd99ae3bd5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:35:47.119869 2740 ts_tablet_manager.cc:1905] T 38d27dc611d64fc4bb42ce26cac089fa P 1fcac2741ad84a55bc68e5dd99ae3bd5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:35:47.126343 2554 tablet_service.cc:1467] Processing CreateTablet for tablet de192567167148bbae44d3e59e20e7db (DEFAULT_TABLE table=test-table [id=7d669c484a754765b1d5947fc7d03ce6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:35:47.126830 2554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de192567167148bbae44d3e59e20e7db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:35:47.127105 2741 ts_tablet_manager.cc:1918] T 38d27dc611d64fc4bb42ce26cac089fa P fa4063ffe74c4ef183d404cf1d2c20cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:35:47.127256 2741 log.cc:1196] T 38d27dc611d64fc4bb42ce26cac089fa P fa4063ffe74c4ef183d404cf1d2c20cd: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/ts-2/wal/wals/38d27dc611d64fc4bb42ce26cac089fa
I20260620 23:35:47.127485 2740 ts_tablet_manager.cc:1918] T 38d27dc611d64fc4bb42ce26cac089fa P 1fcac2741ad84a55bc68e5dd99ae3bd5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:35:47.127637 2740 log.cc:1196] T 38d27dc611d64fc4bb42ce26cac089fa P 1fcac2741ad84a55bc68e5dd99ae3bd5: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/ts-0/wal/wals/38d27dc611d64fc4bb42ce26cac089fa
I20260620 23:35:47.128472 2415 tablet_service.cc:1467] Processing CreateTablet for tablet de192567167148bbae44d3e59e20e7db (DEFAULT_TABLE table=test-table [id=7d669c484a754765b1d5947fc7d03ce6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:35:47.128494 2276 tablet_service.cc:1467] Processing CreateTablet for tablet de192567167148bbae44d3e59e20e7db (DEFAULT_TABLE table=test-table [id=7d669c484a754765b1d5947fc7d03ce6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:35:47.128899 2276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de192567167148bbae44d3e59e20e7db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:35:47.128949 2415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de192567167148bbae44d3e59e20e7db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:35:47.135790 2742 ts_tablet_manager.cc:1905] T 38d27dc611d64fc4bb42ce26cac089fa P f10804aa6b1d4b22a5a82fb8e2e9d0d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:35:47.136198 2753 tablet_bootstrap.cc:492] T de192567167148bbae44d3e59e20e7db P fa4063ffe74c4ef183d404cf1d2c20cd: Bootstrap starting.
I20260620 23:35:47.136693 2751 tablet_bootstrap.cc:492] T de192567167148bbae44d3e59e20e7db P f10804aa6b1d4b22a5a82fb8e2e9d0d6: Bootstrap starting.
I20260620 23:35:47.138571 2753 tablet_bootstrap.cc:654] T de192567167148bbae44d3e59e20e7db P fa4063ffe74c4ef183d404cf1d2c20cd: Neither blocks nor log segments found. Creating new log.
I20260620 23:35:47.138825 2751 tablet_bootstrap.cc:654] T de192567167148bbae44d3e59e20e7db P f10804aa6b1d4b22a5a82fb8e2e9d0d6: Neither blocks nor log segments found. Creating new log.
I20260620 23:35:47.142477 2742 ts_tablet_manager.cc:1918] T 38d27dc611d64fc4bb42ce26cac089fa P f10804aa6b1d4b22a5a82fb8e2e9d0d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:35:47.142833 2742 log.cc:1196] T 38d27dc611d64fc4bb42ce26cac089fa P f10804aa6b1d4b22a5a82fb8e2e9d0d6: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/ts-1/wal/wals/38d27dc611d64fc4bb42ce26cac089fa
I20260620 23:35:47.143440 2741 ts_tablet_manager.cc:1939] T 38d27dc611d64fc4bb42ce26cac089fa P fa4063ffe74c4ef183d404cf1d2c20cd: Deleting consensus metadata
I20260620 23:35:47.143599 2742 ts_tablet_manager.cc:1939] T 38d27dc611d64fc4bb42ce26cac089fa P f10804aa6b1d4b22a5a82fb8e2e9d0d6: Deleting consensus metadata
I20260620 23:35:47.144434 2752 tablet_bootstrap.cc:492] T de192567167148bbae44d3e59e20e7db P 1fcac2741ad84a55bc68e5dd99ae3bd5: Bootstrap starting.
I20260620 23:35:47.145469 2126 catalog_manager.cc:4841] TS f10804aa6b1d4b22a5a82fb8e2e9d0d6 (127.0.97.2:33641): tablet 38d27dc611d64fc4bb42ce26cac089fa (table test-table [id=afe2ff961b6e4480b9a2614390fcf155]) successfully deleted
I20260620 23:35:47.145967 2740 ts_tablet_manager.cc:1939] T 38d27dc611d64fc4bb42ce26cac089fa P 1fcac2741ad84a55bc68e5dd99ae3bd5: Deleting consensus metadata
I20260620 23:35:47.146219 2752 tablet_bootstrap.cc:654] T de192567167148bbae44d3e59e20e7db P 1fcac2741ad84a55bc68e5dd99ae3bd5: Neither blocks nor log segments found. Creating new log.
I20260620 23:35:47.147486 2123 catalog_manager.cc:4841] TS fa4063ffe74c4ef183d404cf1d2c20cd (127.0.97.3:37077): tablet 38d27dc611d64fc4bb42ce26cac089fa (table test-table [id=afe2ff961b6e4480b9a2614390fcf155]) successfully deleted
I20260620 23:35:47.147867 2123 catalog_manager.cc:4841] TS 1fcac2741ad84a55bc68e5dd99ae3bd5 (127.0.97.1:40819): tablet 38d27dc611d64fc4bb42ce26cac089fa (table test-table [id=afe2ff961b6e4480b9a2614390fcf155]) successfully deleted
I20260620 23:35:47.149055 2751 tablet_bootstrap.cc:492] T de192567167148bbae44d3e59e20e7db P f10804aa6b1d4b22a5a82fb8e2e9d0d6: No bootstrap required, opened a new log
I20260620 23:35:47.149261 2751 ts_tablet_manager.cc:1397] T de192567167148bbae44d3e59e20e7db P f10804aa6b1d4b22a5a82fb8e2e9d0d6: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.003s
I20260620 23:35:47.150040 2751 raft_consensus.cc:348] T de192567167148bbae44d3e59e20e7db P f10804aa6b1d4b22a5a82fb8e2e9d0d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40819 } } peers { permanent_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33641 } } peers { permanent_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37077 } }
I20260620 23:35:47.150327 2751 raft_consensus.cc:374] T de192567167148bbae44d3e59e20e7db P f10804aa6b1d4b22a5a82fb8e2e9d0d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:35:47.150434 2751 raft_consensus.cc:729] T de192567167148bbae44d3e59e20e7db P f10804aa6b1d4b22a5a82fb8e2e9d0d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f10804aa6b1d4b22a5a82fb8e2e9d0d6, State: Initialized, Role: FOLLOWER
I20260620 23:35:47.150736 2751 consensus_queue.cc:260] T de192567167148bbae44d3e59e20e7db P f10804aa6b1d4b22a5a82fb8e2e9d0d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40819 } } peers { permanent_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33641 } } peers { permanent_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37077 } }
I20260620 23:35:47.151320 2751 ts_tablet_manager.cc:1428] T de192567167148bbae44d3e59e20e7db P f10804aa6b1d4b22a5a82fb8e2e9d0d6: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260620 23:35:47.153190 2753 tablet_bootstrap.cc:492] T de192567167148bbae44d3e59e20e7db P fa4063ffe74c4ef183d404cf1d2c20cd: No bootstrap required, opened a new log
I20260620 23:35:47.153364 2753 ts_tablet_manager.cc:1397] T de192567167148bbae44d3e59e20e7db P fa4063ffe74c4ef183d404cf1d2c20cd: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.007s
I20260620 23:35:47.153514 2752 tablet_bootstrap.cc:492] T de192567167148bbae44d3e59e20e7db P 1fcac2741ad84a55bc68e5dd99ae3bd5: No bootstrap required, opened a new log
I20260620 23:35:47.153679 2752 ts_tablet_manager.cc:1397] T de192567167148bbae44d3e59e20e7db P 1fcac2741ad84a55bc68e5dd99ae3bd5: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260620 23:35:47.154150 2753 raft_consensus.cc:348] T de192567167148bbae44d3e59e20e7db P fa4063ffe74c4ef183d404cf1d2c20cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40819 } } peers { permanent_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33641 } } peers { permanent_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37077 } }
I20260620 23:35:47.154382 2753 raft_consensus.cc:374] T de192567167148bbae44d3e59e20e7db P fa4063ffe74c4ef183d404cf1d2c20cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:35:47.154467 2753 raft_consensus.cc:729] T de192567167148bbae44d3e59e20e7db P fa4063ffe74c4ef183d404cf1d2c20cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa4063ffe74c4ef183d404cf1d2c20cd, State: Initialized, Role: FOLLOWER
I20260620 23:35:47.154429 2752 raft_consensus.cc:348] T de192567167148bbae44d3e59e20e7db P 1fcac2741ad84a55bc68e5dd99ae3bd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40819 } } peers { permanent_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33641 } } peers { permanent_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37077 } }
I20260620 23:35:47.154661 2752 raft_consensus.cc:374] T de192567167148bbae44d3e59e20e7db P 1fcac2741ad84a55bc68e5dd99ae3bd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:35:47.154752 2752 raft_consensus.cc:729] T de192567167148bbae44d3e59e20e7db P 1fcac2741ad84a55bc68e5dd99ae3bd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fcac2741ad84a55bc68e5dd99ae3bd5, State: Initialized, Role: FOLLOWER
I20260620 23:35:47.154742 2753 consensus_queue.cc:260] T de192567167148bbae44d3e59e20e7db P fa4063ffe74c4ef183d404cf1d2c20cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40819 } } peers { permanent_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33641 } } peers { permanent_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37077 } }
I20260620 23:35:47.155287 2753 ts_tablet_manager.cc:1428] T de192567167148bbae44d3e59e20e7db P fa4063ffe74c4ef183d404cf1d2c20cd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:35:47.155603 2752 consensus_queue.cc:260] T de192567167148bbae44d3e59e20e7db P 1fcac2741ad84a55bc68e5dd99ae3bd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40819 } } peers { permanent_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33641 } } peers { permanent_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37077 } }
I20260620 23:35:47.156241 2752 ts_tablet_manager.cc:1428] T de192567167148bbae44d3e59e20e7db P 1fcac2741ad84a55bc68e5dd99ae3bd5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:35:47.390517 2717 raft_consensus.cc:482] T de192567167148bbae44d3e59e20e7db P fa4063ffe74c4ef183d404cf1d2c20cd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:35:47.390766 2717 raft_consensus.cc:504] T de192567167148bbae44d3e59e20e7db P fa4063ffe74c4ef183d404cf1d2c20cd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40819 } } peers { permanent_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33641 } } peers { permanent_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37077 } }
I20260620 23:35:47.391499 2717 leader_election.cc:288] T de192567167148bbae44d3e59e20e7db P fa4063ffe74c4ef183d404cf1d2c20cd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1fcac2741ad84a55bc68e5dd99ae3bd5 (127.0.97.1:40819), f10804aa6b1d4b22a5a82fb8e2e9d0d6 (127.0.97.2:33641)
I20260620 23:35:47.392493 2296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de192567167148bbae44d3e59e20e7db" candidate_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" is_pre_election: true
I20260620 23:35:47.392618 2434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de192567167148bbae44d3e59e20e7db" candidate_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" is_pre_election: true
I20260620 23:35:47.392808 2296 raft_consensus.cc:2454] T de192567167148bbae44d3e59e20e7db P 1fcac2741ad84a55bc68e5dd99ae3bd5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa4063ffe74c4ef183d404cf1d2c20cd in term 0.
I20260620 23:35:47.392860 2434 raft_consensus.cc:2454] T de192567167148bbae44d3e59e20e7db P f10804aa6b1d4b22a5a82fb8e2e9d0d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa4063ffe74c4ef183d404cf1d2c20cd in term 0.
I20260620 23:35:47.393309 2508 leader_election.cc:302] T de192567167148bbae44d3e59e20e7db P fa4063ffe74c4ef183d404cf1d2c20cd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f10804aa6b1d4b22a5a82fb8e2e9d0d6, fa4063ffe74c4ef183d404cf1d2c20cd; no voters:
I20260620 23:35:47.393581 2717 raft_consensus.cc:2781] T de192567167148bbae44d3e59e20e7db P fa4063ffe74c4ef183d404cf1d2c20cd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:35:47.393698 2717 raft_consensus.cc:482] T de192567167148bbae44d3e59e20e7db P fa4063ffe74c4ef183d404cf1d2c20cd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:35:47.393813 2717 raft_consensus.cc:3037] T de192567167148bbae44d3e59e20e7db P fa4063ffe74c4ef183d404cf1d2c20cd [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:35:47.395865 2717 raft_consensus.cc:504] T de192567167148bbae44d3e59e20e7db P fa4063ffe74c4ef183d404cf1d2c20cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40819 } } peers { permanent_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33641 } } peers { permanent_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37077 } }
I20260620 23:35:47.396595 2717 leader_election.cc:288] T de192567167148bbae44d3e59e20e7db P fa4063ffe74c4ef183d404cf1d2c20cd [CANDIDATE]: Term 1 election: Requested vote from peers 1fcac2741ad84a55bc68e5dd99ae3bd5 (127.0.97.1:40819), f10804aa6b1d4b22a5a82fb8e2e9d0d6 (127.0.97.2:33641)
I20260620 23:35:47.397023 2434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de192567167148bbae44d3e59e20e7db" candidate_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6"
I20260620 23:35:47.397258 2434 raft_consensus.cc:3037] T de192567167148bbae44d3e59e20e7db P f10804aa6b1d4b22a5a82fb8e2e9d0d6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:35:47.397271 2296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de192567167148bbae44d3e59e20e7db" candidate_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5"
I20260620 23:35:47.397513 2296 raft_consensus.cc:3037] T de192567167148bbae44d3e59e20e7db P 1fcac2741ad84a55bc68e5dd99ae3bd5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:35:47.400390 2434 raft_consensus.cc:2454] T de192567167148bbae44d3e59e20e7db P f10804aa6b1d4b22a5a82fb8e2e9d0d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa4063ffe74c4ef183d404cf1d2c20cd in term 1.
I20260620 23:35:47.400672 2296 raft_consensus.cc:2454] T de192567167148bbae44d3e59e20e7db P 1fcac2741ad84a55bc68e5dd99ae3bd5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa4063ffe74c4ef183d404cf1d2c20cd in term 1.
I20260620 23:35:47.400978 2508 leader_election.cc:302] T de192567167148bbae44d3e59e20e7db P fa4063ffe74c4ef183d404cf1d2c20cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f10804aa6b1d4b22a5a82fb8e2e9d0d6, fa4063ffe74c4ef183d404cf1d2c20cd; no voters:
I20260620 23:35:47.401320 2717 raft_consensus.cc:2781] T de192567167148bbae44d3e59e20e7db P fa4063ffe74c4ef183d404cf1d2c20cd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:35:47.401762 2717 raft_consensus.cc:686] T de192567167148bbae44d3e59e20e7db P fa4063ffe74c4ef183d404cf1d2c20cd [term 1 LEADER]: Becoming Leader. State: Replica: fa4063ffe74c4ef183d404cf1d2c20cd, State: Running, Role: LEADER
I20260620 23:35:47.402374 2717 consensus_queue.cc:237] T de192567167148bbae44d3e59e20e7db P fa4063ffe74c4ef183d404cf1d2c20cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40819 } } peers { permanent_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33641 } } peers { permanent_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37077 } }
I20260620 23:35:47.405370 2139 catalog_manager.cc:5495] T de192567167148bbae44d3e59e20e7db P fa4063ffe74c4ef183d404cf1d2c20cd reported cstate change: term changed from 0 to 1, leader changed from <none> to fa4063ffe74c4ef183d404cf1d2c20cd (127.0.97.3). New cstate: current_term: 1 leader_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33641 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa4063ffe74c4ef183d404cf1d2c20cd" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37077 } health_report { overall_health: HEALTHY } } }
I20260620 23:35:47.450067 2773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:47.441357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60262 (local address 127.0.97.3:37077)
0620 23:35:47.444266 (+ 2909us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:47.444272 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:47.444300 (+ 28us) server_negotiation.cc:407] Connection header received
0620 23:35:47.444372 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:47.444378 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:47.444472 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:47.444590 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:35:47.444988 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:47.446146 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:47.448050 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:47.448689 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:47.448835 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:47.449271 (+ 436us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:35:47.449412 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:35:47.449510 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:47.449645 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 23:35:47.449735 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2767,"thread_start_us":2634,"threads_started":1}
I20260620 23:35:47.460436 2296 raft_consensus.cc:1261] T de192567167148bbae44d3e59e20e7db P 1fcac2741ad84a55bc68e5dd99ae3bd5 [term 1 FOLLOWER]: Refusing update from remote peer fa4063ffe74c4ef183d404cf1d2c20cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:35:47.460585 2434 raft_consensus.cc:1261] T de192567167148bbae44d3e59e20e7db P f10804aa6b1d4b22a5a82fb8e2e9d0d6 [term 1 FOLLOWER]: Refusing update from remote peer fa4063ffe74c4ef183d404cf1d2c20cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:35:47.461032 2771 consensus_queue.cc:1035] T de192567167148bbae44d3e59e20e7db P fa4063ffe74c4ef183d404cf1d2c20cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fcac2741ad84a55bc68e5dd99ae3bd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:35:47.461407 2717 consensus_queue.cc:1035] T de192567167148bbae44d3e59e20e7db P fa4063ffe74c4ef183d404cf1d2c20cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f10804aa6b1d4b22a5a82fb8e2e9d0d6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 23:35:47.489677 2722 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 23:35:49.078225 2139 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:35:49.078549 2139 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:35:49.082882 2139 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 78b3f3598f0540858739235691fbe934: Sending DeleteTablet for 3 replicas of tablet de192567167148bbae44d3e59e20e7db
I20260620 23:35:49.083981 2415 tablet_service.cc:1514] Processing DeleteTablet for tablet de192567167148bbae44d3e59e20e7db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:35:49 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37478
I20260620 23:35:49.083973 2554 tablet_service.cc:1514] Processing DeleteTablet for tablet de192567167148bbae44d3e59e20e7db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:35:49 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44508
I20260620 23:35:49.084478 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 2210
I20260620 23:35:49.084488 2276 tablet_service.cc:1514] Processing DeleteTablet for tablet de192567167148bbae44d3e59e20e7db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:35:49 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47690
I20260620 23:35:49.085618 2801 tablet_replica.cc:330] T de192567167148bbae44d3e59e20e7db P fa4063ffe74c4ef183d404cf1d2c20cd: stopping tablet replica
I20260620 23:35:49.085898 2801 raft_consensus.cc:2229] T de192567167148bbae44d3e59e20e7db P fa4063ffe74c4ef183d404cf1d2c20cd [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:35:49.086272 2801 raft_consensus.cc:2258] T de192567167148bbae44d3e59e20e7db P fa4063ffe74c4ef183d404cf1d2c20cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:35:49.086871 2802 tablet_replica.cc:330] T de192567167148bbae44d3e59e20e7db P f10804aa6b1d4b22a5a82fb8e2e9d0d6: stopping tablet replica
I20260620 23:35:49.087575 2801 ts_tablet_manager.cc:1905] T de192567167148bbae44d3e59e20e7db P fa4063ffe74c4ef183d404cf1d2c20cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:35:49.092103 2802 raft_consensus.cc:2229] T de192567167148bbae44d3e59e20e7db P f10804aa6b1d4b22a5a82fb8e2e9d0d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:35:49.092329 2803 tablet_replica.cc:330] T de192567167148bbae44d3e59e20e7db P 1fcac2741ad84a55bc68e5dd99ae3bd5: stopping tablet replica
I20260620 23:35:49.092475 2802 raft_consensus.cc:2258] T de192567167148bbae44d3e59e20e7db P f10804aa6b1d4b22a5a82fb8e2e9d0d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:35:49.092578 2803 raft_consensus.cc:2229] T de192567167148bbae44d3e59e20e7db P 1fcac2741ad84a55bc68e5dd99ae3bd5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:35:49.092864 2803 raft_consensus.cc:2258] T de192567167148bbae44d3e59e20e7db P 1fcac2741ad84a55bc68e5dd99ae3bd5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:35:49.093672 2802 ts_tablet_manager.cc:1905] T de192567167148bbae44d3e59e20e7db P f10804aa6b1d4b22a5a82fb8e2e9d0d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:35:49.093734 2803 ts_tablet_manager.cc:1905] T de192567167148bbae44d3e59e20e7db P 1fcac2741ad84a55bc68e5dd99ae3bd5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:35:49.099743 2803 ts_tablet_manager.cc:1918] T de192567167148bbae44d3e59e20e7db P 1fcac2741ad84a55bc68e5dd99ae3bd5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 23:35:49.099789 2802 ts_tablet_manager.cc:1918] T de192567167148bbae44d3e59e20e7db P f10804aa6b1d4b22a5a82fb8e2e9d0d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 23:35:49.099776 2801 ts_tablet_manager.cc:1918] T de192567167148bbae44d3e59e20e7db P fa4063ffe74c4ef183d404cf1d2c20cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 23:35:49.099936 2803 log.cc:1196] T de192567167148bbae44d3e59e20e7db P 1fcac2741ad84a55bc68e5dd99ae3bd5: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/ts-0/wal/wals/de192567167148bbae44d3e59e20e7db
I20260620 23:35:49.099941 2802 log.cc:1196] T de192567167148bbae44d3e59e20e7db P f10804aa6b1d4b22a5a82fb8e2e9d0d6: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/ts-1/wal/wals/de192567167148bbae44d3e59e20e7db
I20260620 23:35:49.099941 2801 log.cc:1196] T de192567167148bbae44d3e59e20e7db P fa4063ffe74c4ef183d404cf1d2c20cd: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998525501163-388-0/minicluster-data/ts-2/wal/wals/de192567167148bbae44d3e59e20e7db
I20260620 23:35:49.100375 2803 ts_tablet_manager.cc:1939] T de192567167148bbae44d3e59e20e7db P 1fcac2741ad84a55bc68e5dd99ae3bd5: Deleting consensus metadata
I20260620 23:35:49.100428 2801 ts_tablet_manager.cc:1939] T de192567167148bbae44d3e59e20e7db P fa4063ffe74c4ef183d404cf1d2c20cd: Deleting consensus metadata
I20260620 23:35:49.100476 2802 ts_tablet_manager.cc:1939] T de192567167148bbae44d3e59e20e7db P f10804aa6b1d4b22a5a82fb8e2e9d0d6: Deleting consensus metadata
I20260620 23:35:49.101493 2123 catalog_manager.cc:4841] TS 1fcac2741ad84a55bc68e5dd99ae3bd5 (127.0.97.1:40819): tablet de192567167148bbae44d3e59e20e7db (table test-table [id=7d669c484a754765b1d5947fc7d03ce6]) successfully deleted
I20260620 23:35:49.101576 2126 catalog_manager.cc:4841] TS f10804aa6b1d4b22a5a82fb8e2e9d0d6 (127.0.97.2:33641): tablet de192567167148bbae44d3e59e20e7db (table test-table [id=7d669c484a754765b1d5947fc7d03ce6]) successfully deleted
I20260620 23:35:49.101801 2123 catalog_manager.cc:4841] TS fa4063ffe74c4ef183d404cf1d2c20cd (127.0.97.3:37077): tablet de192567167148bbae44d3e59e20e7db (table test-table [id=7d669c484a754765b1d5947fc7d03ce6]) successfully deleted
Jun 20 23:35:49 dist-test-slave-tcgx krb5kdc[2094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998546, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 23:35:49.113097 2805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:49.085829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53444 (local address 127.0.97.1:40819)
0620 23:35:49.091422 (+ 5593us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:49.091429 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:49.092211 (+ 782us) server_negotiation.cc:407] Connection header received
0620 23:35:49.092298 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:49.092303 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:49.092389 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:49.092503 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:35:49.094426 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:49.095671 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:49.097439 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:49.098130 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:49.098273 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:49.106723 (+ 8450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:35:49.106738 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:35:49.106755 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:35:49.106803 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:35:49.109978 (+ 3175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:49.110828 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:49.110840 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:49.110843 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:49.111173 (+ 330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:49.111632 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:49.111636 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:49.111639 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:49.112023 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:35:49.112232 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:49.112630 (+ 398us) server_negotiation.cc:299] Negotiation successful
0620 23:35:49.112813 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5432,"thread_start_us":5309,"threads_started":1}
W20260620 23:35:49.113960 2341 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53444
W20260620 23:35:49.114710 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 2210: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:35:49.114909 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 2210
I20260620 23:35:49.141743 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 2350
Jun 20 23:35:49 dist-test-slave-tcgx krb5kdc[2094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998546, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 23:35:49.157140 2806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:49.142489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43286 (local address 127.0.97.2:33641)
0620 23:35:49.143081 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:49.143086 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:49.143102 (+ 16us) server_negotiation.cc:407] Connection header received
0620 23:35:49.143155 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:49.143158 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:49.143218 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:49.143310 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:35:49.144489 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:49.145235 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:49.146422 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:49.146810 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:49.146907 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:49.151248 (+ 4341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:35:49.151265 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:35:49.151280 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:35:49.151321 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:35:49.154333 (+ 3012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:49.155073 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:49.155081 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:49.155084 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:49.155224 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:49.155597 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:49.155603 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:49.155606 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:49.155966 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:35:49.156281 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:49.156573 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 23:35:49.156799 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":303,"threads_started":1}
W20260620 23:35:49.157912 2480 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43286
W20260620 23:35:49.158586 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 2350: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:35:49.158775 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 2350
I20260620 23:35:49.181381 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 2489
Jun 20 23:35:49 dist-test-slave-tcgx krb5kdc[2094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998546, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 23:35:49.197958 2807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:49.182186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60278 (local address 127.0.97.3:37077)
0620 23:35:49.182711 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:49.182716 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:49.182731 (+ 15us) server_negotiation.cc:407] Connection header received
0620 23:35:49.182783 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:49.182786 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:49.182844 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:49.182994 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:35:49.184326 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:49.185089 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:49.186355 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:49.186747 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:49.186887 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:49.191775 (+ 4888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:35:49.191792 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:35:49.191809 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:35:49.191852 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:35:49.194916 (+ 3064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:49.195791 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:49.195807 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:49.195809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:49.195945 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:49.196341 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:49.196346 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:49.196351 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:49.196741 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:35:49.197036 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:49.197368 (+ 332us) server_negotiation.cc:299] Negotiation successful
0620 23:35:49.197629 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":264,"threads_started":1}
W20260620 23:35:49.198606 2619 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60278
W20260620 23:35:49.199293 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 2489: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:35:49.199442 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 2489
I20260620 23:35:49.221733 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 2107
I20260620 23:35:49.223066 2179 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:35:49.338665 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 2107
2026-06-20T23:35:49Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8827 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:35:49 dist-test-slave-tcgx krb5kdc[2811](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:35:49 dist-test-slave-tcgx krb5kdc[2811](info): set up 2 sockets
Jun 20 23:35:49 dist-test-slave-tcgx krb5kdc[2811](info): commencing operation
krb5kdc: starting...
W20260620 23:35:50.487366 388 mini_kdc.cc:121] Time spent starting KDC: real 1.101s user 0.002s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/krb5kdc/test-user.keytab.
Jun 20 23:35:50 dist-test-slave-tcgx krb5kdc[2811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:35:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:35:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 23:35:50.682858 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32923
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:32923
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:35:51.006809 2824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:35:51.007184 2824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:35:51.007278 2824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:35:51.014607 2824 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:35:51.014734 2824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:35:51.014782 2824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:35:51.014839 2824 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:35:51.014884 2824 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:35:51.025017 2824 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:32923
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:35:51.026394 2824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:35:51.027825 2824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:35:51.040102 2830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:35:51.040205 2829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:35:51.040217 2832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:35:51.040869 2824 server_base.cc:1034] running on GCE node
I20260620 23:35:51.041846 2824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:35:51.043465 2824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:35:51.044683 2824 hybrid_clock.cc:648] HybridClock initialized: now 1781998551044644 us; error 33 us; skew 500 ppm
I20260620 23:35:51.045113 2824 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:35:51 dist-test-slave-tcgx krb5kdc[2811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998551, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:35:51.048709 2824 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 23:35:51.050351 2824 webserver.cc:460] Webserver started at http://127.0.97.62:41395/ using document root <none> and password file <none>
I20260620 23:35:51.051201 2824 fs_manager.cc:359] Metadata directory not provided
I20260620 23:35:51.051358 2824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:35:51.051692 2824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:35:51.055013 2824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/minicluster-data/master-0/data/instance:
uuid: "81a54314a80940d08b2c9516ac85ad45"
format_stamp: "Formatted at 2026-06-20 23:35:51 on dist-test-slave-tcgx"
server_key: "7db3f07b77f0f352cdea8608c9227723"
server_key_iv: "926f750e7ea1433b1ac9b51f7de8fd4a"
server_key_version: "encryptionkey@0"
I20260620 23:35:51.055867 2824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/minicluster-data/master-0/wal/instance:
uuid: "81a54314a80940d08b2c9516ac85ad45"
format_stamp: "Formatted at 2026-06-20 23:35:51 on dist-test-slave-tcgx"
server_key: "7db3f07b77f0f352cdea8608c9227723"
server_key_iv: "926f750e7ea1433b1ac9b51f7de8fd4a"
server_key_version: "encryptionkey@0"
I20260620 23:35:51.061344 2824 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:35:51.064941 2839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:35:51.066334 2824 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:35:51.066538 2824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/minicluster-data/master-0/wal
uuid: "81a54314a80940d08b2c9516ac85ad45"
format_stamp: "Formatted at 2026-06-20 23:35:51 on dist-test-slave-tcgx"
server_key: "7db3f07b77f0f352cdea8608c9227723"
server_key_iv: "926f750e7ea1433b1ac9b51f7de8fd4a"
server_key_version: "encryptionkey@0"
I20260620 23:35:51.066740 2824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:35:51.081624 2824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:35:51.086328 2824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:35:51.086632 2824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:35:51.105782 2824 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:32923
I20260620 23:35:51.105863 2890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:32923 every 8 connection(s)
I20260620 23:35:51.107316 2824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:35:51.111372 2891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:35:51.111840 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 2824
I20260620 23:35:51.112033 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/minicluster-data/master-0/wal/instance
I20260620 23:35:51.112699 388 external_mini_cluster.cc:1428] Setting key 5799da515ddad978e7c0ac22e3085d09
I20260620 23:35:51.126350 2891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81a54314a80940d08b2c9516ac85ad45: Bootstrap starting.
Jun 20 23:35:51 dist-test-slave-tcgx krb5kdc[2811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:35:51.129901 2891 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81a54314a80940d08b2c9516ac85ad45: Neither blocks nor log segments found. Creating new log.
I20260620 23:35:51.131390 2891 log.cc:824] T 00000000000000000000000000000000 P 81a54314a80940d08b2c9516ac85ad45: Log is configured to *not* fsync() on all Append() calls
I20260620 23:35:51.135174 2891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81a54314a80940d08b2c9516ac85ad45: No bootstrap required, opened a new log
I20260620 23:35:51.139276 2894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:51.114924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50862 (local address 127.0.97.62:32923)
0620 23:35:51.115854 (+ 930us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:51.115878 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:51.115937 (+ 59us) server_negotiation.cc:407] Connection header received
0620 23:35:51.118283 (+ 2346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:51.118320 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:51.118954 (+ 634us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:51.119469 (+ 515us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:35:51.120948 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:51.123059 (+ 2111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:51.124434 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:51.125520 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:51.125705 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:51.129913 (+ 4208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:35:51.129927 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:35:51.129952 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:35:51.130020 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:35:51.134115 (+ 4095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:51.134977 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:51.134985 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:51.134994 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:51.135226 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:51.135641 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:51.135648 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:51.135652 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:51.136373 (+ 721us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:35:51.136650 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:51.137050 (+ 400us) server_negotiation.cc:299] Negotiation successful
0620 23:35:51.137764 (+ 714us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":356,"threads_started":1}
I20260620 23:35:51.144475 2891 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81a54314a80940d08b2c9516ac85ad45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81a54314a80940d08b2c9516ac85ad45" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32923 } }
I20260620 23:35:51.144836 2891 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81a54314a80940d08b2c9516ac85ad45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:35:51.144963 2891 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81a54314a80940d08b2c9516ac85ad45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81a54314a80940d08b2c9516ac85ad45, State: Initialized, Role: FOLLOWER
I20260620 23:35:51.145588 2891 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81a54314a80940d08b2c9516ac85ad45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81a54314a80940d08b2c9516ac85ad45" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32923 } }
I20260620 23:35:51.145829 2891 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81a54314a80940d08b2c9516ac85ad45 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:35:51.145973 2891 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81a54314a80940d08b2c9516ac85ad45 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:35:51.146143 2891 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81a54314a80940d08b2c9516ac85ad45 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:35:51.148480 2891 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81a54314a80940d08b2c9516ac85ad45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81a54314a80940d08b2c9516ac85ad45" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32923 } }
I20260620 23:35:51.148996 2891 leader_election.cc:302] T 00000000000000000000000000000000 P 81a54314a80940d08b2c9516ac85ad45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81a54314a80940d08b2c9516ac85ad45; no voters:
I20260620 23:35:51.149570 2891 leader_election.cc:288] T 00000000000000000000000000000000 P 81a54314a80940d08b2c9516ac85ad45 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:35:51.149748 2896 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81a54314a80940d08b2c9516ac85ad45 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:35:51.150828 2896 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81a54314a80940d08b2c9516ac85ad45 [term 1 LEADER]: Becoming Leader. State: Replica: 81a54314a80940d08b2c9516ac85ad45, State: Running, Role: LEADER
I20260620 23:35:51.151490 2896 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81a54314a80940d08b2c9516ac85ad45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81a54314a80940d08b2c9516ac85ad45" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32923 } }
I20260620 23:35:51.151957 2891 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81a54314a80940d08b2c9516ac85ad45 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:35:51.154863 2897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81a54314a80940d08b2c9516ac85ad45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81a54314a80940d08b2c9516ac85ad45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81a54314a80940d08b2c9516ac85ad45" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32923 } } }
I20260620 23:35:51.154927 2898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81a54314a80940d08b2c9516ac85ad45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81a54314a80940d08b2c9516ac85ad45. Latest consensus state: current_term: 1 leader_uuid: "81a54314a80940d08b2c9516ac85ad45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81a54314a80940d08b2c9516ac85ad45" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32923 } } }
I20260620 23:35:51.155289 2897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81a54314a80940d08b2c9516ac85ad45 [sys.catalog]: This master's current role is: LEADER
I20260620 23:35:51.155309 2898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81a54314a80940d08b2c9516ac85ad45 [sys.catalog]: This master's current role is: LEADER
W20260620 23:35:51.167470 2912 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 81a54314a80940d08b2c9516ac85ad45: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:35:51.167649 2912 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:35:51.167825 2907 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:35:51.170336 2907 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:35:51.179554 2907 catalog_manager.cc:1347] Generated new cluster ID: 422d49b9ab494ea8af6fcbe468ae1848
I20260620 23:35:51.179708 2907 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:35:51.188972 2907 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:35:51.191118 2907 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:35:51.202188 2907 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81a54314a80940d08b2c9516ac85ad45: Generated new TSK 0
I20260620 23:35:51.203442 2907 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 23:35:51.260761 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32923
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:35:51.591809 2917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:35:51.592166 2917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:35:51.592322 2917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:35:51.599736 2917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:35:51.599882 2917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:35:51.600064 2917 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 23:35:51.610143 2917 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32923
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:35:51.611552 2917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:35:51.612875 2917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:35:51.625844 2922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:35:51.625883 2923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:35:51.625803 2925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:35:51.631049 2917 server_base.cc:1034] running on GCE node
I20260620 23:35:51.631675 2917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:35:51.632658 2917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:35:51.633855 2917 hybrid_clock.cc:648] HybridClock initialized: now 1781998551633811 us; error 47 us; skew 500 ppm
I20260620 23:35:51.634140 2917 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:35:51 dist-test-slave-tcgx krb5kdc[2811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998551, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:35:51.637310 2917 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 23:35:51.638875 2917 webserver.cc:460] Webserver started at http://127.0.97.1:38771/ using document root <none> and password file <none>
I20260620 23:35:51.639644 2917 fs_manager.cc:359] Metadata directory not provided
I20260620 23:35:51.639797 2917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:35:51.640161 2917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:35:51.643093 2917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-0/data/instance:
uuid: "730f94d418254831812c020fcd9c1acf"
format_stamp: "Formatted at 2026-06-20 23:35:51 on dist-test-slave-tcgx"
server_key: "d224c20e7dd67c099f8ab5e7eb9f9266"
server_key_iv: "bb0f2639266e3df770a658598a3a8201"
server_key_version: "encryptionkey@0"
I20260620 23:35:51.643847 2917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-0/wal/instance:
uuid: "730f94d418254831812c020fcd9c1acf"
format_stamp: "Formatted at 2026-06-20 23:35:51 on dist-test-slave-tcgx"
server_key: "d224c20e7dd67c099f8ab5e7eb9f9266"
server_key_iv: "bb0f2639266e3df770a658598a3a8201"
server_key_version: "encryptionkey@0"
I20260620 23:35:51.648984 2917 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 23:35:51.652570 2932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:35:51.654119 2917 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 23:35:51.654263 2917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-0/wal
uuid: "730f94d418254831812c020fcd9c1acf"
format_stamp: "Formatted at 2026-06-20 23:35:51 on dist-test-slave-tcgx"
server_key: "d224c20e7dd67c099f8ab5e7eb9f9266"
server_key_iv: "bb0f2639266e3df770a658598a3a8201"
server_key_version: "encryptionkey@0"
I20260620 23:35:51.654477 2917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:35:51.672125 2917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:35:51.676739 2917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:35:51.676992 2917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:35:51.677939 2917 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:35:51.679369 2917 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:35:51.679445 2917 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:35:51.679541 2917 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:35:51.679584 2917 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:35:51.713053 2917 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38769
I20260620 23:35:51.713096 3044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38769 every 8 connection(s)
I20260620 23:35:51.714407 2917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
I20260620 23:35:51.721612 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 2917
I20260620 23:35:51.721834 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-0/wal/instance
I20260620 23:35:51.722456 388 external_mini_cluster.cc:1428] Setting key f80ee82457fc5623b5a09fcdc1b5b84c
Jun 20 23:35:51 dist-test-slave-tcgx krb5kdc[2811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998551, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:35:51.737003 3047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:51.717290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46327 (local address 127.0.97.62:32923)
0620 23:35:51.717763 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:51.717768 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:51.718841 (+ 1073us) server_negotiation.cc:407] Connection header received
0620 23:35:51.720856 (+ 2015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:51.720862 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:51.720960 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:51.721080 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:35:51.722896 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:51.723797 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:51.725762 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:51.726570 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:51.726720 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:51.730558 (+ 3838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:35:51.730564 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:35:51.730567 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:35:51.730602 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:35:51.733232 (+ 2630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:51.734310 (+ 1078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:51.734314 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:51.734317 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:51.734443 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:51.735030 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:51.735033 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:51.735036 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:51.735304 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:35:51.735415 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:51.736436 (+ 1021us) server_negotiation.cc:299] Negotiation successful
0620 23:35:51.736623 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":238,"threads_started":1}
I20260620 23:35:51.738704 3048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:51.717887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32923 (local address 127.0.97.1:46327)
0620 23:35:51.718641 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0620 23:35:51.718682 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 23:35:51.720531 (+ 1849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:35:51.721356 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:35:51.721366 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:35:51.722023 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:35:51.722700 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:51.722726 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:51.723990 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:51.723994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:51.725501 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:51.725516 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:51.726829 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:51.726835 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:51.727153 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:51.727908 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:35:51.727936 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:35:51.730286 (+ 2350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:35:51.733407 (+ 3121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:35:51.733420 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:35:51.733445 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:35:51.734172 (+ 727us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:35:51.734565 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:35:51.734573 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:35:51.734579 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:35:51.734858 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:35:51.735443 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:35:51.735455 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:35:51.735991 (+ 536us) client_negotiation.cc:769] Sending connection context
0620 23:35:51.736340 (+ 349us) client_negotiation.cc:240] Negotiation successful
0620 23:35:51.736999 (+ 659us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":289,"threads_started":1}
I20260620 23:35:51.740600 3045 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32923
I20260620 23:35:51.740938 3045 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 23:35:51.741753 3045 heartbeater.cc:511] Master 127.0.97.62:32923 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260620 23:35:51.744151 2856 ts_manager.cc:194] Registered new tserver with Master: 730f94d418254831812c020fcd9c1acf (127.0.97.1:38769)
I20260620 23:35:51.745620 2856 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:46327
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 23:35:51.767993 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32923
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:35:52.097992 3051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:35:52.098300 3051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:35:52.098454 3051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:35:52.106395 3051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:35:52.106545 3051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:35:52.106705 3051 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 23:35:52.122396 3051 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32923
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:35:52.124101 3051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:35:52.125793 3051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:35:52.140003 3059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:35:52.140069 3056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:35:52.140038 3057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:35:52.140889 3051 server_base.cc:1034] running on GCE node
I20260620 23:35:52.141541 3051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:35:52.142557 3051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:35:52.143810 3051 hybrid_clock.cc:648] HybridClock initialized: now 1781998552143755 us; error 44 us; skew 500 ppm
I20260620 23:35:52.144261 3051 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:35:52 dist-test-slave-tcgx krb5kdc[2811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998552, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:35:52.147843 3051 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 23:35:52.149565 3051 webserver.cc:460] Webserver started at http://127.0.97.2:41801/ using document root <none> and password file <none>
I20260620 23:35:52.150338 3051 fs_manager.cc:359] Metadata directory not provided
I20260620 23:35:52.150483 3051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:35:52.150835 3051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:35:52.154367 3051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-1/data/instance:
uuid: "29b2c49c7c9d40af8934a1f6209f3007"
format_stamp: "Formatted at 2026-06-20 23:35:52 on dist-test-slave-tcgx"
server_key: "e0d7b3d8991e74403dfcaebeaff05200"
server_key_iv: "eaf13f96210fa7dbb4e4122dc5bc3d40"
server_key_version: "encryptionkey@0"
I20260620 23:35:52.155426 3051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-1/wal/instance:
uuid: "29b2c49c7c9d40af8934a1f6209f3007"
format_stamp: "Formatted at 2026-06-20 23:35:52 on dist-test-slave-tcgx"
server_key: "e0d7b3d8991e74403dfcaebeaff05200"
server_key_iv: "eaf13f96210fa7dbb4e4122dc5bc3d40"
server_key_version: "encryptionkey@0"
I20260620 23:35:52.161352 3051 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 23:35:52.165126 3066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:35:52.166576 3051 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:35:52.166764 3051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-1/wal
uuid: "29b2c49c7c9d40af8934a1f6209f3007"
format_stamp: "Formatted at 2026-06-20 23:35:52 on dist-test-slave-tcgx"
server_key: "e0d7b3d8991e74403dfcaebeaff05200"
server_key_iv: "eaf13f96210fa7dbb4e4122dc5bc3d40"
server_key_version: "encryptionkey@0"
I20260620 23:35:52.166996 3051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:35:52.186031 3051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:35:52.190496 3051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:35:52.190757 3051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:35:52.191795 3051 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:35:52.193183 3051 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:35:52.193262 3051 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:35:52.193351 3051 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:35:52.193394 3051 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:35:52.227362 3051 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39541
I20260620 23:35:52.227418 3178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39541 every 8 connection(s)
I20260620 23:35:52.228740 3051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
I20260620 23:35:52.237586 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 3051
I20260620 23:35:52.237771 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-1/wal/instance
I20260620 23:35:52.238204 388 external_mini_cluster.cc:1428] Setting key cafd99f2b3345e6a17d6849485da782a
Jun 20 23:35:52 dist-test-slave-tcgx krb5kdc[2811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998552, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:35:52.252341 3047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:52.231395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42917 (local address 127.0.97.62:32923)
0620 23:35:52.231601 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:52.231606 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:52.232886 (+ 1280us) server_negotiation.cc:407] Connection header received
0620 23:35:52.234958 (+ 2072us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:52.234966 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:52.235063 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:52.235198 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:35:52.237022 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:52.237932 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:52.240027 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:52.240499 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:52.240602 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:52.245330 (+ 4728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:35:52.245341 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:35:52.245350 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:35:52.245403 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:35:52.248624 (+ 3221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:52.249665 (+ 1041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:52.249669 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:52.249672 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:52.249809 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:52.250361 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:52.250365 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:52.250367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:52.250654 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:35:52.250780 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:52.251891 (+ 1111us) server_negotiation.cc:299] Negotiation successful
0620 23:35:52.252106 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 23:35:52.253943 3181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:52.231915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32923 (local address 127.0.97.2:42917)
0620 23:35:52.232679 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0620 23:35:52.232735 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 23:35:52.234611 (+ 1876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:35:52.235462 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:35:52.235471 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:35:52.236089 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:35:52.236841 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:52.236860 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:52.238094 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:52.238099 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:52.239864 (+ 1765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:52.239884 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:52.240755 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:52.240763 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:52.241133 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:52.241945 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:35:52.241993 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:35:52.245040 (+ 3047us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:35:52.248804 (+ 3764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:35:52.248817 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:35:52.248841 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:35:52.249526 (+ 685us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:35:52.249941 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:35:52.249949 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:35:52.249955 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:35:52.250218 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:35:52.250800 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:35:52.250811 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:35:52.251352 (+ 541us) client_negotiation.cc:769] Sending connection context
0620 23:35:52.251683 (+ 331us) client_negotiation.cc:240] Negotiation successful
0620 23:35:52.252320 (+ 637us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":297,"threads_started":1}
I20260620 23:35:52.255815 3179 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32923
I20260620 23:35:52.256173 3179 heartbeater.cc:464] Registering TS with master...
I20260620 23:35:52.256999 3179 heartbeater.cc:511] Master 127.0.97.62:32923 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 23:35:52.258720 2856 ts_manager.cc:194] Registered new tserver with Master: 29b2c49c7c9d40af8934a1f6209f3007 (127.0.97.2:39541)
I20260620 23:35:52.259792 2856 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:42917
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 23:35:52.284250 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32923
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:35:52.611586 3184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:35:52.611923 3184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:35:52.612087 3184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:35:52.619346 3184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:35:52.619517 3184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:35:52.619699 3184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 23:35:52.629725 3184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32923
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:35:52.631142 3184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:35:52.632516 3184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:35:52.645848 3190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:35:52.645887 3192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:35:52.646106 3189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:35:52.646304 3184 server_base.cc:1034] running on GCE node
I20260620 23:35:52.647173 3184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:35:52.648124 3184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:35:52.649353 3184 hybrid_clock.cc:648] HybridClock initialized: now 1781998552649306 us; error 40 us; skew 500 ppm
I20260620 23:35:52.649668 3184 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:35:52 dist-test-slave-tcgx krb5kdc[2811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998552, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:35:52.652962 3184 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 23:35:52.654592 3184 webserver.cc:460] Webserver started at http://127.0.97.3:39673/ using document root <none> and password file <none>
I20260620 23:35:52.655454 3184 fs_manager.cc:359] Metadata directory not provided
I20260620 23:35:52.655582 3184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:35:52.655897 3184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:35:52.659083 3184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-2/data/instance:
uuid: "ab7f8386a2874b7daa8ce4fc204dc5f5"
format_stamp: "Formatted at 2026-06-20 23:35:52 on dist-test-slave-tcgx"
server_key: "e3307291d75896b410c5ba58fde290a0"
server_key_iv: "5e73568f3c06cf27a578399538c17f5b"
server_key_version: "encryptionkey@0"
I20260620 23:35:52.659909 3184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-2/wal/instance:
uuid: "ab7f8386a2874b7daa8ce4fc204dc5f5"
format_stamp: "Formatted at 2026-06-20 23:35:52 on dist-test-slave-tcgx"
server_key: "e3307291d75896b410c5ba58fde290a0"
server_key_iv: "5e73568f3c06cf27a578399538c17f5b"
server_key_version: "encryptionkey@0"
I20260620 23:35:52.665134 3184 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 23:35:52.668622 3199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:35:52.669987 3184 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 23:35:52.670167 3184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-2/wal
uuid: "ab7f8386a2874b7daa8ce4fc204dc5f5"
format_stamp: "Formatted at 2026-06-20 23:35:52 on dist-test-slave-tcgx"
server_key: "e3307291d75896b410c5ba58fde290a0"
server_key_iv: "5e73568f3c06cf27a578399538c17f5b"
server_key_version: "encryptionkey@0"
I20260620 23:35:52.670387 3184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:35:52.688879 3184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:35:52.693447 3184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:35:52.693759 3184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:35:52.694757 3184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:35:52.696233 3184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:35:52.696316 3184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:35:52.696463 3184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:35:52.696512 3184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:35:52.729940 3184 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34711
I20260620 23:35:52.730048 3311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34711 every 8 connection(s)
I20260620 23:35:52.731277 3184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
I20260620 23:35:52.732540 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 3184
I20260620 23:35:52.732713 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-2/wal/instance
I20260620 23:35:52.733314 388 external_mini_cluster.cc:1428] Setting key c91a58bbfd72bc9e3aef9072d7c8ba8a
Jun 20 23:35:52 dist-test-slave-tcgx krb5kdc[2811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998552, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:35:52.748965 3045 heartbeater.cc:503] Master 127.0.97.62:32923 was elected leader, sending a full tablet report...
I20260620 23:35:52.753561 3047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:52.734339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43029 (local address 127.0.97.62:32923)
0620 23:35:52.734545 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:52.734551 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:52.736081 (+ 1530us) server_negotiation.cc:407] Connection header received
0620 23:35:52.738269 (+ 2188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:52.738275 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:52.738375 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:52.738486 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:35:52.740176 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:52.741012 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:52.742737 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:52.743224 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:52.743364 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:52.747052 (+ 3688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:35:52.747057 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:35:52.747061 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:35:52.747093 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:35:52.750314 (+ 3221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:52.751187 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:52.751191 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:52.751194 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:52.751340 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:52.751750 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:52.751754 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:52.751756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:52.752054 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:35:52.752263 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:52.753067 (+ 804us) server_negotiation.cc:299] Negotiation successful
0620 23:35:52.753299 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 23:35:52.754561 3314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:52.735042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32923 (local address 127.0.97.3:43029)
0620 23:35:52.735924 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0620 23:35:52.735973 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 23:35:52.737979 (+ 2006us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:35:52.738743 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:35:52.738752 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:35:52.739373 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:35:52.739982 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:52.740000 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:52.741196 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:52.741201 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:52.742569 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:52.742581 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:52.743488 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:52.743496 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:52.743753 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:52.744502 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:35:52.744529 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:35:52.746763 (+ 2234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:35:52.750474 (+ 3711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:35:52.750485 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:35:52.750506 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:35:52.751060 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:35:52.751442 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:35:52.751446 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:35:52.751448 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:35:52.751648 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:35:52.752330 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:35:52.752339 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:35:52.752715 (+ 376us) client_negotiation.cc:769] Sending connection context
0620 23:35:52.752949 (+ 234us) client_negotiation.cc:240] Negotiation successful
0620 23:35:52.753384 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":391,"threads_started":1}
I20260620 23:35:52.756229 3312 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32923
I20260620 23:35:52.756513 3312 heartbeater.cc:464] Registering TS with master...
I20260620 23:35:52.757361 3312 heartbeater.cc:511] Master 127.0.97.62:32923 requested a full tablet report, sending...
I20260620 23:35:52.758975 2856 ts_manager.cc:194] Registered new tserver with Master: ab7f8386a2874b7daa8ce4fc204dc5f5 (127.0.97.3:34711)
I20260620 23:35:52.759792 2856 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:43029
I20260620 23:35:52.761942 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 23:35:52.793037 3047 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:35:52.790488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50872 (local address 127.0.97.62:32923)
0620 23:35:52.790757 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:52.790766 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:52.791143 (+ 377us) server_negotiation.cc:407] Connection header received
0620 23:35:52.791359 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:52.791363 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:52.791426 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:52.791530 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:35:52.792767 (+ 1237us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50872: BlockingRecv error: recv got EOF from 127.0.0.1:50872 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 23:35:52.796193 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 2917
W20260620 23:35:52.799376 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 2917: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:38769: server requires authentication, but client does not have Kerberos credentials available
I20260620 23:35:52.799566 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 2917
I20260620 23:35:52.816470 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 3051
W20260620 23:35:52.819312 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 3051: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:39541: server requires authentication, but client does not have Kerberos credentials available
I20260620 23:35:52.819471 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 3051
I20260620 23:35:52.834600 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 3184
W20260620 23:35:52.837550 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 3184: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:34711: server requires authentication, but client does not have Kerberos credentials available
I20260620 23:35:52.837728 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 3184
I20260620 23:35:52.852963 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 2824
I20260620 23:35:52.854272 2886 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:35:52.956321 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 2824
2026-06-20T23:35:52Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3612 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781998525501163-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:35:53 dist-test-slave-tcgx krb5kdc[3329](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:35:53 dist-test-slave-tcgx krb5kdc[3329](info): set up 2 sockets
Jun 20 23:35:53 dist-test-slave-tcgx krb5kdc[3329](info): commencing operation
krb5kdc: starting...
W20260620 23:35:54.055310 388 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781998525501163-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781998525501163-388-0/krb5kdc/test-user.keytab.
Jun 20 23:35:54 dist-test-slave-tcgx krb5kdc[3329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:35:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:35:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 23:35:54.251260 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781998525501163-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781998525501163-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781998525501163-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41047
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41047
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:35:54.586571 3342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:35:54.587071 3342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:35:54.587247 3342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:35:54.595961 3342 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:35:54.596117 3342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:35:54.596223 3342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:35:54.596324 3342 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:35:54.596405 3342 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:35:54.607096 3342 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781998525501163-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781998525501163-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41047
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781998525501163-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:35:54.608552 3342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:35:54.609928 3342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:35:54.621634 3350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:35:54.621567 3347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:35:54.621604 3348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:35:54.622256 3342 server_base.cc:1034] running on GCE node
I20260620 23:35:54.623095 3342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:35:54.624531 3342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:35:54.625735 3342 hybrid_clock.cc:648] HybridClock initialized: now 1781998554625693 us; error 39 us; skew 500 ppm
I20260620 23:35:54.626081 3342 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:35:54 dist-test-slave-tcgx krb5kdc[3329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998554, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:35:54.635092 3342 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 23:35:54.636778 3342 webserver.cc:460] Webserver started at http://127.0.97.62:44343/ using document root <none> and password file <none>
I20260620 23:35:54.637478 3342 fs_manager.cc:359] Metadata directory not provided
I20260620 23:35:54.637581 3342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:35:54.637840 3342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:35:54.640892 3342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781998525501163-388-0/minicluster-data/master-0/data/instance:
uuid: "2fe00bbd00744a258488fee995e14186"
format_stamp: "Formatted at 2026-06-20 23:35:54 on dist-test-slave-tcgx"
server_key: "400d03906a3148f53e6a89e69c50bf17"
server_key_iv: "86838dafa6c83ed702f56083e79418f4"
server_key_version: "encryptionkey@0"
I20260620 23:35:54.641649 3342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781998525501163-388-0/minicluster-data/master-0/wal/instance:
uuid: "2fe00bbd00744a258488fee995e14186"
format_stamp: "Formatted at 2026-06-20 23:35:54 on dist-test-slave-tcgx"
server_key: "400d03906a3148f53e6a89e69c50bf17"
server_key_iv: "86838dafa6c83ed702f56083e79418f4"
server_key_version: "encryptionkey@0"
I20260620 23:35:54.646833 3342 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 23:35:54.650729 3357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:35:54.652140 3342 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:35:54.652361 3342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781998525501163-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781998525501163-388-0/minicluster-data/master-0/wal
uuid: "2fe00bbd00744a258488fee995e14186"
format_stamp: "Formatted at 2026-06-20 23:35:54 on dist-test-slave-tcgx"
server_key: "400d03906a3148f53e6a89e69c50bf17"
server_key_iv: "86838dafa6c83ed702f56083e79418f4"
server_key_version: "encryptionkey@0"
I20260620 23:35:54.652547 3342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781998525501163-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781998525501163-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781998525501163-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:35:54.681110 3342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:35:54.685669 3342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:35:54.685963 3342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:35:54.704437 3342 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41047
I20260620 23:35:54.704469 3408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41047 every 8 connection(s)
I20260620 23:35:54.705853 3342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:35:54.709514 3409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:35:54.710099 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 3342
I20260620 23:35:54.710310 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781998525501163-388-0/minicluster-data/master-0/wal/instance
I20260620 23:35:54.710913 388 external_mini_cluster.cc:1428] Setting key 6a2729ba401b62df1440a3ccb67a953d
I20260620 23:35:54.723371 3409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fe00bbd00744a258488fee995e14186: Bootstrap starting.
Jun 20 23:35:54 dist-test-slave-tcgx krb5kdc[3329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:35:54.727474 3409 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2fe00bbd00744a258488fee995e14186: Neither blocks nor log segments found. Creating new log.
I20260620 23:35:54.728550 3409 log.cc:824] T 00000000000000000000000000000000 P 2fe00bbd00744a258488fee995e14186: Log is configured to *not* fsync() on all Append() calls
I20260620 23:35:54.731873 3409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fe00bbd00744a258488fee995e14186: No bootstrap required, opened a new log
I20260620 23:35:54.734990 3412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:54.712982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55180 (local address 127.0.97.62:41047)
0620 23:35:54.713774 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:54.713786 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:54.713827 (+ 41us) server_negotiation.cc:407] Connection header received
0620 23:35:54.715494 (+ 1667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:54.715518 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:54.715928 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:54.716304 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:35:54.718070 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:54.719444 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:54.721237 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:54.721916 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:54.722070 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:54.727030 (+ 4960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:35:54.727042 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:35:54.727058 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:35:54.727101 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:35:54.730653 (+ 3552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:54.731562 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:54.731569 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:54.731576 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:54.731757 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:54.732169 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:54.732172 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:54.732174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:54.732774 (+ 600us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:35:54.732999 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:54.733462 (+ 463us) server_negotiation.cc:299] Negotiation successful
0620 23:35:54.733848 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":376,"threads_started":1}
I20260620 23:35:54.738910 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:35:54.740295 3409 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2fe00bbd00744a258488fee995e14186 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fe00bbd00744a258488fee995e14186" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41047 } }
I20260620 23:35:54.740658 3409 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2fe00bbd00744a258488fee995e14186 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:35:54.740775 3409 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2fe00bbd00744a258488fee995e14186 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fe00bbd00744a258488fee995e14186, State: Initialized, Role: FOLLOWER
I20260620 23:35:54.741362 3409 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2fe00bbd00744a258488fee995e14186 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fe00bbd00744a258488fee995e14186" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41047 } }
I20260620 23:35:54.741582 3409 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2fe00bbd00744a258488fee995e14186 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:35:54.741700 3409 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2fe00bbd00744a258488fee995e14186 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:35:54.741868 3409 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2fe00bbd00744a258488fee995e14186 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:35:54.744074 3409 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2fe00bbd00744a258488fee995e14186 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fe00bbd00744a258488fee995e14186" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41047 } }
I20260620 23:35:54.744643 3409 leader_election.cc:302] T 00000000000000000000000000000000 P 2fe00bbd00744a258488fee995e14186 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2fe00bbd00744a258488fee995e14186; no voters:
I20260620 23:35:54.745353 3409 leader_election.cc:288] T 00000000000000000000000000000000 P 2fe00bbd00744a258488fee995e14186 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:35:54.745463 3415 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2fe00bbd00744a258488fee995e14186 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:35:54.747226 3415 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2fe00bbd00744a258488fee995e14186 [term 1 LEADER]: Becoming Leader. State: Replica: 2fe00bbd00744a258488fee995e14186, State: Running, Role: LEADER
I20260620 23:35:54.747884 3409 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2fe00bbd00744a258488fee995e14186 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:35:54.747803 3415 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2fe00bbd00744a258488fee995e14186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fe00bbd00744a258488fee995e14186" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41047 } }
I20260620 23:35:54.750365 3417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fe00bbd00744a258488fee995e14186 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2fe00bbd00744a258488fee995e14186. Latest consensus state: current_term: 1 leader_uuid: "2fe00bbd00744a258488fee995e14186" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fe00bbd00744a258488fee995e14186" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41047 } } }
I20260620 23:35:54.750754 3417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fe00bbd00744a258488fee995e14186 [sys.catalog]: This master's current role is: LEADER
I20260620 23:35:54.751325 3416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fe00bbd00744a258488fee995e14186 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2fe00bbd00744a258488fee995e14186" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fe00bbd00744a258488fee995e14186" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41047 } } }
I20260620 23:35:54.751617 3416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fe00bbd00744a258488fee995e14186 [sys.catalog]: This master's current role is: LEADER
I20260620 23:35:54.762414 3420 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:35:54.769910 3420 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:35:54.778774 3433 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 23:35:54.781985 3420 catalog_manager.cc:1347] Generated new cluster ID: adafd582243b4eebb41b8bdc75f627ed
I20260620 23:35:54.782110 3420 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:35:54.791893 3412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:54.782037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55186 (local address 127.0.97.62:41047)
0620 23:35:54.782234 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:54.782240 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:54.783305 (+ 1065us) server_negotiation.cc:407] Connection header received
0620 23:35:54.783514 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:54.783520 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:54.783602 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:54.783722 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:35:54.784888 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:54.786032 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:54.787832 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:54.788600 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:54.788737 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:54.789923 (+ 1186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:35:54.789930 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:35:54.789935 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:35:54.790858 (+ 923us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:35:54.790961 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:35:54.790998 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:35:54.791116 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:54.791240 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 23:35:54.791351 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 23:35:54.802160 3420 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:35:54.803565 3420 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:35:54.817662 3420 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2fe00bbd00744a258488fee995e14186: Generated new TSK 0
I20260620 23:35:54.818524 3420 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:35:54.831658 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 3342
I20260620 23:35:54.832849 3404 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:35:54.924003 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 3342
2026-06-20T23:35:54Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1962 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:35:55 dist-test-slave-tcgx krb5kdc[3443](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:35:55 dist-test-slave-tcgx krb5kdc[3443](info): set up 2 sockets
Jun 20 23:35:55 dist-test-slave-tcgx krb5kdc[3443](info): commencing operation
krb5kdc: starting...
W20260620 23:35:56.007429 388 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.005s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/krb5kdc/test-user.keytab.
Jun 20 23:35:56 dist-test-slave-tcgx krb5kdc[3443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:35:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:35:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 23:35:56.206207 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37225
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37225
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:35:56.531983 3456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:35:56.532302 3456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:35:56.532471 3456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:35:56.539773 3456 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:35:56.539897 3456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:35:56.539979 3456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:35:56.540083 3456 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:35:56.540169 3456 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:35:56.550163 3456 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37225
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:35:56.551627 3456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:35:56.552979 3456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:35:56.564857 3461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:35:56.564796 3464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:35:56.564884 3462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:35:56.565762 3456 server_base.cc:1034] running on GCE node
I20260620 23:35:56.566593 3456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:35:56.568143 3456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:35:56.569379 3456 hybrid_clock.cc:648] HybridClock initialized: now 1781998556569318 us; error 52 us; skew 500 ppm
I20260620 23:35:56.569741 3456 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:35:56 dist-test-slave-tcgx krb5kdc[3443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:35:56.572803 3456 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 23:35:56.574326 3456 webserver.cc:460] Webserver started at http://127.0.97.62:44453/ using document root <none> and password file <none>
I20260620 23:35:56.575095 3456 fs_manager.cc:359] Metadata directory not provided
I20260620 23:35:56.575233 3456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:35:56.575534 3456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:35:56.578511 3456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/minicluster-data/master-0/data/instance:
uuid: "3ef858053a6e4ab1886456dcfa2617ba"
format_stamp: "Formatted at 2026-06-20 23:35:56 on dist-test-slave-tcgx"
server_key: "0ad8aa6db72c2b6fbf611d459411761b"
server_key_iv: "8f4231ecb55a0e3c2d5e380312c4deeb"
server_key_version: "encryptionkey@0"
I20260620 23:35:56.579341 3456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/minicluster-data/master-0/wal/instance:
uuid: "3ef858053a6e4ab1886456dcfa2617ba"
format_stamp: "Formatted at 2026-06-20 23:35:56 on dist-test-slave-tcgx"
server_key: "0ad8aa6db72c2b6fbf611d459411761b"
server_key_iv: "8f4231ecb55a0e3c2d5e380312c4deeb"
server_key_version: "encryptionkey@0"
I20260620 23:35:56.584664 3456 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 23:35:56.588300 3471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:35:56.589717 3456 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:35:56.589931 3456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/minicluster-data/master-0/wal
uuid: "3ef858053a6e4ab1886456dcfa2617ba"
format_stamp: "Formatted at 2026-06-20 23:35:56 on dist-test-slave-tcgx"
server_key: "0ad8aa6db72c2b6fbf611d459411761b"
server_key_iv: "8f4231ecb55a0e3c2d5e380312c4deeb"
server_key_version: "encryptionkey@0"
I20260620 23:35:56.590140 3456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:35:56.624385 3456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:35:56.628741 3456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:35:56.629055 3456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:35:56.647352 3456 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37225
I20260620 23:35:56.647375 3522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37225 every 8 connection(s)
I20260620 23:35:56.649029 3456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:35:56.652834 3523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:35:56.655242 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 3456
I20260620 23:35:56.655472 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/minicluster-data/master-0/wal/instance
I20260620 23:35:56.656245 388 external_mini_cluster.cc:1428] Setting key 20f280479d060145954b376fbe3b5c31
I20260620 23:35:56.664618 3523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ef858053a6e4ab1886456dcfa2617ba: Bootstrap starting.
I20260620 23:35:56.667788 3523 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ef858053a6e4ab1886456dcfa2617ba: Neither blocks nor log segments found. Creating new log.
I20260620 23:35:56.668781 3523 log.cc:824] T 00000000000000000000000000000000 P 3ef858053a6e4ab1886456dcfa2617ba: Log is configured to *not* fsync() on all Append() calls
Jun 20 23:35:56 dist-test-slave-tcgx krb5kdc[3443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:35:56.671975 3523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ef858053a6e4ab1886456dcfa2617ba: No bootstrap required, opened a new log
I20260620 23:35:56.679404 3526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:56.658491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53948 (local address 127.0.97.62:37225)
0620 23:35:56.659333 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:56.659346 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:56.659386 (+ 40us) server_negotiation.cc:407] Connection header received
0620 23:35:56.661099 (+ 1713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:56.661123 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:56.661522 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:56.661937 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:35:56.663359 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:56.664598 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:56.665885 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:56.666535 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:56.666744 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:56.670769 (+ 4025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:35:56.670780 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:35:56.670799 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:35:56.670844 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:35:56.674741 (+ 3897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:56.675497 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:56.675505 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:56.675514 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:56.675719 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:56.676128 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:56.676133 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:56.676136 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:56.676824 (+ 688us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:35:56.677140 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:56.677511 (+ 371us) server_negotiation.cc:299] Negotiation successful
0620 23:35:56.677975 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":378,"threads_started":1}
I20260620 23:35:56.682474 3523 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ef858053a6e4ab1886456dcfa2617ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ef858053a6e4ab1886456dcfa2617ba" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37225 } }
I20260620 23:35:56.682818 3523 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ef858053a6e4ab1886456dcfa2617ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:35:56.682893 3523 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ef858053a6e4ab1886456dcfa2617ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ef858053a6e4ab1886456dcfa2617ba, State: Initialized, Role: FOLLOWER
I20260620 23:35:56.683512 3523 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ef858053a6e4ab1886456dcfa2617ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ef858053a6e4ab1886456dcfa2617ba" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37225 } }
I20260620 23:35:56.683719 3523 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ef858053a6e4ab1886456dcfa2617ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:35:56.683857 3523 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ef858053a6e4ab1886456dcfa2617ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:35:56.683983 3523 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ef858053a6e4ab1886456dcfa2617ba [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:35:56.686641 3523 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ef858053a6e4ab1886456dcfa2617ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ef858053a6e4ab1886456dcfa2617ba" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37225 } }
I20260620 23:35:56.687391 3523 leader_election.cc:302] T 00000000000000000000000000000000 P 3ef858053a6e4ab1886456dcfa2617ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ef858053a6e4ab1886456dcfa2617ba; no voters:
I20260620 23:35:56.688109 3523 leader_election.cc:288] T 00000000000000000000000000000000 P 3ef858053a6e4ab1886456dcfa2617ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:35:56.688234 3528 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ef858053a6e4ab1886456dcfa2617ba [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:35:56.688935 3528 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ef858053a6e4ab1886456dcfa2617ba [term 1 LEADER]: Becoming Leader. State: Replica: 3ef858053a6e4ab1886456dcfa2617ba, State: Running, Role: LEADER
I20260620 23:35:56.689502 3528 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ef858053a6e4ab1886456dcfa2617ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ef858053a6e4ab1886456dcfa2617ba" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37225 } }
I20260620 23:35:56.690374 3523 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ef858053a6e4ab1886456dcfa2617ba [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:35:56.692833 3530 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ef858053a6e4ab1886456dcfa2617ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ef858053a6e4ab1886456dcfa2617ba. Latest consensus state: current_term: 1 leader_uuid: "3ef858053a6e4ab1886456dcfa2617ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ef858053a6e4ab1886456dcfa2617ba" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37225 } } }
I20260620 23:35:56.692898 3529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ef858053a6e4ab1886456dcfa2617ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ef858053a6e4ab1886456dcfa2617ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ef858053a6e4ab1886456dcfa2617ba" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37225 } } }
I20260620 23:35:56.693243 3530 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ef858053a6e4ab1886456dcfa2617ba [sys.catalog]: This master's current role is: LEADER
I20260620 23:35:56.693300 3529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ef858053a6e4ab1886456dcfa2617ba [sys.catalog]: This master's current role is: LEADER
I20260620 23:35:56.695144 3535 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:35:56.700529 3535 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:35:56.709834 3535 catalog_manager.cc:1347] Generated new cluster ID: 7c990df4104548bfbae3f55f807da027
I20260620 23:35:56.709971 3535 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:35:56.722344 3535 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:35:56.723721 3535 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:35:56.730700 3535 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ef858053a6e4ab1886456dcfa2617ba: Generated new TSK 0
I20260620 23:35:56.731590 3535 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 23:35:56.791173 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37225
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:35:57.121995 3549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:35:57.122361 3549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:35:57.122536 3549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:35:57.129977 3549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:35:57.130160 3549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:35:57.130347 3549 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 23:35:57.140336 3549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:35:57.141716 3549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:35:57.143074 3549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:35:57.155846 3555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:35:57.155910 3554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:35:57.155879 3557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:35:57.156579 3549 server_base.cc:1034] running on GCE node
I20260620 23:35:57.157145 3549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:35:57.158026 3549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:35:57.159286 3549 hybrid_clock.cc:648] HybridClock initialized: now 1781998557159242 us; error 33 us; skew 500 ppm
I20260620 23:35:57.159626 3549 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:35:57 dist-test-slave-tcgx krb5kdc[3443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998557, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:35:57.163007 3549 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 23:35:57.164484 3549 webserver.cc:460] Webserver started at http://127.0.97.1:39691/ using document root <none> and password file <none>
I20260620 23:35:57.165186 3549 fs_manager.cc:359] Metadata directory not provided
I20260620 23:35:57.165318 3549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:35:57.165612 3549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:35:57.168618 3549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-0/data/instance:
uuid: "0388a764303745989e0ba7b450daaef7"
format_stamp: "Formatted at 2026-06-20 23:35:57 on dist-test-slave-tcgx"
server_key: "9ab68b24d575d615dd5271844e045cab"
server_key_iv: "4a599c37f54f63e8dfa1689f60537ee1"
server_key_version: "encryptionkey@0"
I20260620 23:35:57.169394 3549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-0/wal/instance:
uuid: "0388a764303745989e0ba7b450daaef7"
format_stamp: "Formatted at 2026-06-20 23:35:57 on dist-test-slave-tcgx"
server_key: "9ab68b24d575d615dd5271844e045cab"
server_key_iv: "4a599c37f54f63e8dfa1689f60537ee1"
server_key_version: "encryptionkey@0"
I20260620 23:35:57.174503 3549 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 23:35:57.178123 3564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:35:57.179764 3549 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:35:57.179971 3549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-0/wal
uuid: "0388a764303745989e0ba7b450daaef7"
format_stamp: "Formatted at 2026-06-20 23:35:57 on dist-test-slave-tcgx"
server_key: "9ab68b24d575d615dd5271844e045cab"
server_key_iv: "4a599c37f54f63e8dfa1689f60537ee1"
server_key_version: "encryptionkey@0"
I20260620 23:35:57.180204 3549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:35:57.209877 3549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:35:57.214221 3549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:35:57.214524 3549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:35:57.215557 3549 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:35:57.216887 3549 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:35:57.216990 3549 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:35:57.217133 3549 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:35:57.217222 3549 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:35:57.249770 3549 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43579
I20260620 23:35:57.249814 3676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43579 every 8 connection(s)
I20260620 23:35:57.251171 3549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
I20260620 23:35:57.252936 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 3549
I20260620 23:35:57.253165 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-0/wal/instance
I20260620 23:35:57.253839 388 external_mini_cluster.cc:1428] Setting key b09ca10eff5ffc3ff7785bae642e7681
Jun 20 23:35:57 dist-test-slave-tcgx krb5kdc[3443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998557, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 23:35:57.275491 3679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:57.253845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49233 (local address 127.0.97.62:37225)
0620 23:35:57.254334 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:57.254349 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:57.255439 (+ 1090us) server_negotiation.cc:407] Connection header received
0620 23:35:57.258189 (+ 2750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:57.258193 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:57.258268 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:57.258365 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:35:57.260587 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:57.261802 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:57.263597 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:57.264389 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:57.264535 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:57.268280 (+ 3745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:35:57.268289 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:35:57.268297 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:35:57.268342 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:35:57.272037 (+ 3695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:57.272800 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:57.272807 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:57.272810 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:57.272959 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:57.273398 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:57.273406 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:57.273409 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:57.273818 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:35:57.273991 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:57.274714 (+ 723us) server_negotiation.cc:299] Negotiation successful
0620 23:35:57.274890 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":231,"threads_started":1}
I20260620 23:35:57.276297 3680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:57.254501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37225 (local address 127.0.97.1:49233)
0620 23:35:57.255291 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0620 23:35:57.255335 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 23:35:57.257875 (+ 2540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:35:57.258651 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:35:57.258666 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:35:57.259506 (+ 840us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:35:57.260364 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:57.260393 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:57.262010 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:57.262018 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:57.263450 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:57.263461 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:57.264610 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:57.264615 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:57.264817 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:57.265512 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:35:57.265534 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:35:57.267993 (+ 2459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:35:57.272193 (+ 4200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:35:57.272200 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:35:57.272215 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:35:57.272654 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:35:57.273067 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:35:57.273070 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:35:57.273072 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:35:57.273273 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:35:57.273955 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:35:57.273961 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:35:57.274344 (+ 383us) client_negotiation.cc:769] Sending connection context
0620 23:35:57.274588 (+ 244us) client_negotiation.cc:240] Negotiation successful
0620 23:35:57.275060 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":352,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260620 23:35:57.278038 3677 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37225
I20260620 23:35:57.278353 3677 heartbeater.cc:464] Registering TS with master...
I20260620 23:35:57.279156 3677 heartbeater.cc:511] Master 127.0.97.62:37225 requested a full tablet report, sending...
I20260620 23:35:57.281427 3488 ts_manager.cc:194] Registered new tserver with Master: 0388a764303745989e0ba7b450daaef7 (127.0.97.1:43579)
I20260620 23:35:57.283339 3488 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:49233
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 23:35:57.300457 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37225
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:35:57.633693 3683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:35:57.633989 3683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:35:57.634140 3683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:35:57.641467 3683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:35:57.641618 3683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:35:57.641796 3683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 23:35:57.652341 3683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:35:57.653740 3683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:35:57.655164 3683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:35:57.668288 3689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:35:57.668238 3688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:35:57.668201 3691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:35:57.672564 3683 server_base.cc:1034] running on GCE node
I20260620 23:35:57.673234 3683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:35:57.674220 3683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:35:57.675432 3683 hybrid_clock.cc:648] HybridClock initialized: now 1781998557675373 us; error 58 us; skew 500 ppm
I20260620 23:35:57.675746 3683 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:35:57 dist-test-slave-tcgx krb5kdc[3443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998557, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:35:57.678850 3683 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 23:35:57.680485 3683 webserver.cc:460] Webserver started at http://127.0.97.2:40777/ using document root <none> and password file <none>
I20260620 23:35:57.681232 3683 fs_manager.cc:359] Metadata directory not provided
I20260620 23:35:57.681377 3683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:35:57.681671 3683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:35:57.684826 3683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-1/data/instance:
uuid: "e9e21daa154b492c8bc0184276237fce"
format_stamp: "Formatted at 2026-06-20 23:35:57 on dist-test-slave-tcgx"
server_key: "73824c79f547ad298936e465b61fbf41"
server_key_iv: "553486ec4dfdf66af584f5fbaf7802d9"
server_key_version: "encryptionkey@0"
I20260620 23:35:57.685604 3683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-1/wal/instance:
uuid: "e9e21daa154b492c8bc0184276237fce"
format_stamp: "Formatted at 2026-06-20 23:35:57 on dist-test-slave-tcgx"
server_key: "73824c79f547ad298936e465b61fbf41"
server_key_iv: "553486ec4dfdf66af584f5fbaf7802d9"
server_key_version: "encryptionkey@0"
I20260620 23:35:57.690789 3683 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 23:35:57.694458 3698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:35:57.695963 3683 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:35:57.696151 3683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-1/wal
uuid: "e9e21daa154b492c8bc0184276237fce"
format_stamp: "Formatted at 2026-06-20 23:35:57 on dist-test-slave-tcgx"
server_key: "73824c79f547ad298936e465b61fbf41"
server_key_iv: "553486ec4dfdf66af584f5fbaf7802d9"
server_key_version: "encryptionkey@0"
I20260620 23:35:57.696367 3683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:35:57.711421 3683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:35:57.715539 3683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:35:57.715847 3683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:35:57.716812 3683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:35:57.718226 3683 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:35:57.718326 3683 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:35:57.718472 3683 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:35:57.718545 3683 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:35:57.750574 3683 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36425
I20260620 23:35:57.750622 3810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36425 every 8 connection(s)
I20260620 23:35:57.751950 3683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
I20260620 23:35:57.758802 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 3683
I20260620 23:35:57.759020 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-1/wal/instance
I20260620 23:35:57.759513 388 external_mini_cluster.cc:1428] Setting key 59a86653df6d8703a31cce4f9c35956b
Jun 20 23:35:57 dist-test-slave-tcgx krb5kdc[3443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998557, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:35:57.773468 3679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:57.754591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:32943 (local address 127.0.97.62:37225)
0620 23:35:57.754797 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:57.754803 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:57.756122 (+ 1319us) server_negotiation.cc:407] Connection header received
0620 23:35:57.757873 (+ 1751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:57.757878 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:57.757947 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:57.758038 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:35:57.759719 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:57.760575 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:57.762436 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:57.763277 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:57.763425 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:57.767288 (+ 3863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:35:57.767293 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:35:57.767297 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:35:57.767328 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:35:57.769965 (+ 2637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:57.770995 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:57.771000 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:57.771002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:57.771129 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:57.771684 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:57.771688 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:57.771691 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:57.771985 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:35:57.772111 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:57.773036 (+ 925us) server_negotiation.cc:299] Negotiation successful
0620 23:35:57.773225 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 23:35:57.775208 3813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:57.755227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37225 (local address 127.0.97.2:32943)
0620 23:35:57.755952 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0620 23:35:57.755993 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 23:35:57.757600 (+ 1607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:35:57.758255 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:35:57.758263 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:35:57.758766 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:35:57.759481 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:57.759498 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:57.760727 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:57.760731 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:57.762291 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:57.762302 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:57.763484 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:57.763489 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:57.763689 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:57.764403 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:35:57.764426 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:35:57.767024 (+ 2598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:35:57.770119 (+ 3095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:35:57.770132 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:35:57.770155 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:35:57.770804 (+ 649us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:35:57.771244 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:35:57.771252 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:35:57.771259 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:35:57.771521 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:35:57.772121 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:35:57.772133 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:35:57.772621 (+ 488us) client_negotiation.cc:769] Sending connection context
0620 23:35:57.772945 (+ 324us) client_negotiation.cc:240] Negotiation successful
0620 23:35:57.773581 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":276,"threads_started":1}
I20260620 23:35:57.776737 3811 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37225
I20260620 23:35:57.777065 3811 heartbeater.cc:464] Registering TS with master...
I20260620 23:35:57.777806 3811 heartbeater.cc:511] Master 127.0.97.62:37225 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 23:35:57.779335 3488 ts_manager.cc:194] Registered new tserver with Master: e9e21daa154b492c8bc0184276237fce (127.0.97.2:36425)
I20260620 23:35:57.780226 3488 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:32943
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 23:35:57.802582 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37225
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:35:58.125018 3816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:35:58.125396 3816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:35:58.125560 3816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:35:58.133119 3816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:35:58.133296 3816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:35:58.133476 3816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 23:35:58.143564 3816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:35:58.144960 3816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:35:58.146301 3816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:35:58.159495 3822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:35:58.159747 3824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:35:58.159538 3821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:35:58.160573 3816 server_base.cc:1034] running on GCE node
I20260620 23:35:58.161198 3816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:35:58.162199 3816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:35:58.163415 3816 hybrid_clock.cc:648] HybridClock initialized: now 1781998558163337 us; error 86 us; skew 500 ppm
I20260620 23:35:58.163710 3816 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:35:58 dist-test-slave-tcgx krb5kdc[3443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998558, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:35:58.166893 3816 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 23:35:58.168365 3816 webserver.cc:460] Webserver started at http://127.0.97.3:41577/ using document root <none> and password file <none>
I20260620 23:35:58.169076 3816 fs_manager.cc:359] Metadata directory not provided
I20260620 23:35:58.169205 3816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:35:58.169516 3816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:35:58.172580 3816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-2/data/instance:
uuid: "008664f78f0a4f19934e3397ea3689f2"
format_stamp: "Formatted at 2026-06-20 23:35:58 on dist-test-slave-tcgx"
server_key: "f15d6666cb2229ebeae952497a6921db"
server_key_iv: "8e96762898315cc3ccf1e909d5be428f"
server_key_version: "encryptionkey@0"
I20260620 23:35:58.173367 3816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-2/wal/instance:
uuid: "008664f78f0a4f19934e3397ea3689f2"
format_stamp: "Formatted at 2026-06-20 23:35:58 on dist-test-slave-tcgx"
server_key: "f15d6666cb2229ebeae952497a6921db"
server_key_iv: "8e96762898315cc3ccf1e909d5be428f"
server_key_version: "encryptionkey@0"
I20260620 23:35:58.178712 3816 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260620 23:35:58.182523 3831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:35:58.184274 3816 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:35:58.184474 3816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-2/wal
uuid: "008664f78f0a4f19934e3397ea3689f2"
format_stamp: "Formatted at 2026-06-20 23:35:58 on dist-test-slave-tcgx"
server_key: "f15d6666cb2229ebeae952497a6921db"
server_key_iv: "8e96762898315cc3ccf1e909d5be428f"
server_key_version: "encryptionkey@0"
I20260620 23:35:58.184751 3816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:35:58.224131 3816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:35:58.228750 3816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:35:58.229022 3816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:35:58.229972 3816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:35:58.231518 3816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:35:58.231596 3816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:35:58.231683 3816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:35:58.231727 3816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:35:58.265623 3816 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46041
I20260620 23:35:58.265666 3943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46041 every 8 connection(s)
I20260620 23:35:58.266904 3816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
I20260620 23:35:58.271471 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 3816
I20260620 23:35:58.271687 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998525501163-388-0/minicluster-data/ts-2/wal/instance
I20260620 23:35:58.272200 388 external_mini_cluster.cc:1428] Setting key db774c4ce10803c1c0c3786350430bf1
Jun 20 23:35:58 dist-test-slave-tcgx krb5kdc[3443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998558, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:35:58.286396 3677 heartbeater.cc:503] Master 127.0.97.62:37225 was elected leader, sending a full tablet report...
I20260620 23:35:58.287911 3679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:58.269534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55953 (local address 127.0.97.62:37225)
0620 23:35:58.269709 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:58.269714 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:58.270825 (+ 1111us) server_negotiation.cc:407] Connection header received
0620 23:35:58.273375 (+ 2550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:58.273380 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:58.273456 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:58.273564 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:35:58.275319 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:58.276074 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:58.277551 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:58.277942 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:58.278069 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:58.281435 (+ 3366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:35:58.281440 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:35:58.281443 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:35:58.281474 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:35:58.283821 (+ 2347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:58.284649 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:58.284653 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:58.284655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:58.284774 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:58.285276 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:58.285280 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:58.285284 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:58.285538 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:35:58.285681 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:58.287185 (+ 1504us) server_negotiation.cc:299] Negotiation successful
0620 23:35:58.287461 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 23:35:58.288978 3946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:58.269959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37225 (local address 127.0.97.3:55953)
0620 23:35:58.270670 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0620 23:35:58.270709 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 23:35:58.273064 (+ 2355us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:35:58.273795 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:35:58.273804 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:35:58.274427 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:35:58.275169 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:58.275186 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:58.276217 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:58.276222 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:58.277417 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:35:58.277428 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:35:58.278160 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:35:58.278167 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:35:58.278403 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:58.279074 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:35:58.279096 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:35:58.281225 (+ 2129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:35:58.284003 (+ 2778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:35:58.284012 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:35:58.284051 (+ 39us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:35:58.284517 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:35:58.284900 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:35:58.284905 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:35:58.284909 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:35:58.285163 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:35:58.285666 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:35:58.285674 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:35:58.286110 (+ 436us) client_negotiation.cc:769] Sending connection context
0620 23:35:58.287037 (+ 927us) client_negotiation.cc:240] Negotiation successful
0620 23:35:58.287601 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":279,"threads_started":1}
I20260620 23:35:58.290489 3944 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37225
I20260620 23:35:58.290804 3944 heartbeater.cc:464] Registering TS with master...
I20260620 23:35:58.291643 3944 heartbeater.cc:511] Master 127.0.97.62:37225 requested a full tablet report, sending...
I20260620 23:35:58.292995 3488 ts_manager.cc:194] Registered new tserver with Master: 008664f78f0a4f19934e3397ea3689f2 (127.0.97.3:46041)
I20260620 23:35:58.293782 3488 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:55953
I20260620 23:35:58.300405 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 23:35:58 dist-test-slave-tcgx krb5kdc[3443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998558, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 23:35:58 dist-test-slave-tcgx krb5kdc[3443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998558, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:35:58.348716 3679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:58.334831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53952 (local address 127.0.97.62:37225)
0620 23:35:58.335047 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:58.335052 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:58.335580 (+ 528us) server_negotiation.cc:407] Connection header received
0620 23:35:58.335782 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:58.335786 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:58.335855 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:58.335954 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:35:58.337324 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:58.338048 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:58.339664 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:58.340068 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:58.340168 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:58.344085 (+ 3917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:35:58.344090 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:35:58.344093 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:35:58.344127 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:35:58.346427 (+ 2300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:58.347061 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:58.347065 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:58.347068 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:58.347174 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:58.347558 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:58.347561 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:58.347563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:58.347863 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:35:58.348036 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:58.348318 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 23:35:58.348490 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260620 23:35:58.349257 3488 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:53952
I20260620 23:35:58.351163 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 3549
Jun 20 23:35:58 dist-test-slave-tcgx krb5kdc[3443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998558, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 23:35:58.366456 3955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:58.352310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54954 (local address 127.0.97.1:43579)
0620 23:35:58.352967 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:58.352974 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:58.352998 (+ 24us) server_negotiation.cc:407] Connection header received
0620 23:35:58.353053 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:58.353058 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:58.353279 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:58.353404 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:35:58.354524 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:58.355566 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:58.356700 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:58.357251 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:58.357344 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:58.361301 (+ 3957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:35:58.361313 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:35:58.361328 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:35:58.361361 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:35:58.364080 (+ 2719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:58.364711 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:58.364718 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:58.364720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:58.364830 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:58.365144 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:58.365147 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:58.365149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:58.365482 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:35:58.365697 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:58.365887 (+ 190us) server_negotiation.cc:299] Negotiation successful
0620 23:35:58.366071 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":351,"threads_started":1}
W20260620 23:35:58.367774 3672 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:54954
W20260620 23:35:58.368871 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 3549: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:35:58.369047 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 3549
I20260620 23:35:58.384692 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 3683
Jun 20 23:35:58 dist-test-slave-tcgx krb5kdc[3443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998558, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 23:35:58.401489 3956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:58.385621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54272 (local address 127.0.97.2:36425)
0620 23:35:58.386178 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:58.386185 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:58.386213 (+ 28us) server_negotiation.cc:407] Connection header received
0620 23:35:58.386277 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:58.386283 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:58.386512 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:58.386649 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:35:58.387881 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:58.388917 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:58.390197 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:58.390820 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:58.390918 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:58.395401 (+ 4483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:35:58.395414 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:35:58.395430 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:35:58.395469 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:35:58.398422 (+ 2953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:58.399209 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:58.399218 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:58.399220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:58.399356 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:58.399834 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:58.399838 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:58.399840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:58.400220 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:35:58.400482 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:58.400837 (+ 355us) server_negotiation.cc:299] Negotiation successful
0620 23:35:58.401074 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":295,"threads_started":1}
W20260620 23:35:58.402720 3806 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:54272
W20260620 23:35:58.403900 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 3683: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:35:58.404088 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 3683
I20260620 23:35:58.419401 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 3816
Jun 20 23:35:58 dist-test-slave-tcgx krb5kdc[3443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998558, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 23:35:58.435845 3957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:35:58.420435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60998 (local address 127.0.97.3:46041)
0620 23:35:58.421110 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0620 23:35:58.421117 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:35:58.421146 (+ 29us) server_negotiation.cc:407] Connection header received
0620 23:35:58.421207 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:35:58.421213 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:35:58.421435 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:35:58.421575 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:35:58.422826 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:58.424044 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:58.425251 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:35:58.425830 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:35:58.425932 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:35:58.430229 (+ 4297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:35:58.430243 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:35:58.430258 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:35:58.430301 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:35:58.433144 (+ 2843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:58.433859 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:58.433866 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:58.433868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:58.433992 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:35:58.434291 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:35:58.434294 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:35:58.434296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:35:58.434635 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:35:58.434872 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0620 23:35:58.435236 (+ 364us) server_negotiation.cc:299] Negotiation successful
0620 23:35:58.435452 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":367,"threads_started":1}
W20260620 23:35:58.437094 3939 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:60998
W20260620 23:35:58.438234 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 3816: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:35:58.438411 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 3816
I20260620 23:35:58.453783 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 3456
I20260620 23:35:58.455097 3518 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:35:58.557936 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 3456
2026-06-20T23:35:58Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3639 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:35:58 dist-test-slave-tcgx krb5kdc[3961](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:35:58 dist-test-slave-tcgx krb5kdc[3961](info): set up 2 sockets
Jun 20 23:35:58 dist-test-slave-tcgx krb5kdc[3961](info): commencing operation
krb5kdc: starting...
W20260620 23:35:59.659945 388 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/krb5kdc/test-user.keytab.
Jun 20 23:35:59 dist-test-slave-tcgx krb5kdc[3961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:35:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:35:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 23:35:59.862872 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43533
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43533
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:36:00.188570 3974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:36:00.188905 3974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:36:00.188998 3974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:36:00.197077 3974 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:36:00.197207 3974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:36:00.197257 3974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:36:00.197315 3974 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:36:00.197353 3974 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:36:00.208045 3974 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43533
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:36:00.209445 3974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:36:00.210858 3974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:36:00.223373 3980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:00.223392 3979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:00.223373 3982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:36:00.224489 3974 server_base.cc:1034] running on GCE node
I20260620 23:36:00.225423 3974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:36:00.227095 3974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:36:00.228410 3974 hybrid_clock.cc:648] HybridClock initialized: now 1781998560228360 us; error 62 us; skew 500 ppm
I20260620 23:36:00.228806 3974 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:36:00 dist-test-slave-tcgx krb5kdc[3961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998560, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:36:00.239903 3974 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 23:36:00.241520 3974 webserver.cc:460] Webserver started at http://127.0.97.62:39981/ using document root <none> and password file <none>
I20260620 23:36:00.242347 3974 fs_manager.cc:359] Metadata directory not provided
I20260620 23:36:00.242506 3974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:36:00.242836 3974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:36:00.246194 3974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/minicluster-data/master-0/data/instance:
uuid: "9155caa57ec84308a655a111a9b03c2f"
format_stamp: "Formatted at 2026-06-20 23:36:00 on dist-test-slave-tcgx"
server_key: "b9294f6d53e4a031cdbb9469fbc8c742"
server_key_iv: "27c13fb34d7cf583cecd0a62a30885e9"
server_key_version: "encryptionkey@0"
I20260620 23:36:00.247139 3974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/minicluster-data/master-0/wal/instance:
uuid: "9155caa57ec84308a655a111a9b03c2f"
format_stamp: "Formatted at 2026-06-20 23:36:00 on dist-test-slave-tcgx"
server_key: "b9294f6d53e4a031cdbb9469fbc8c742"
server_key_iv: "27c13fb34d7cf583cecd0a62a30885e9"
server_key_version: "encryptionkey@0"
I20260620 23:36:00.252763 3974 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:36:00.256604 3989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:00.258134 3974 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 23:36:00.258387 3974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/minicluster-data/master-0/wal
uuid: "9155caa57ec84308a655a111a9b03c2f"
format_stamp: "Formatted at 2026-06-20 23:36:00 on dist-test-slave-tcgx"
server_key: "b9294f6d53e4a031cdbb9469fbc8c742"
server_key_iv: "27c13fb34d7cf583cecd0a62a30885e9"
server_key_version: "encryptionkey@0"
I20260620 23:36:00.258612 3974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:36:00.276999 3974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:36:00.281225 3974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:36:00.281522 3974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:36:00.300375 3974 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43533
I20260620 23:36:00.300377 4040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43533 every 8 connection(s)
I20260620 23:36:00.301870 3974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:36:00.302538 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 3974
I20260620 23:36:00.302771 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/minicluster-data/master-0/wal/instance
I20260620 23:36:00.303567 388 external_mini_cluster.cc:1428] Setting key 9303654779ce8a1be791be43d1e2ed68
I20260620 23:36:00.306584 4041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 23:36:00 dist-test-slave-tcgx krb5kdc[3961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:36:00.319753 4041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9155caa57ec84308a655a111a9b03c2f: Bootstrap starting.
I20260620 23:36:00.324738 4041 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9155caa57ec84308a655a111a9b03c2f: Neither blocks nor log segments found. Creating new log.
I20260620 23:36:00.326179 4041 log.cc:824] T 00000000000000000000000000000000 P 9155caa57ec84308a655a111a9b03c2f: Log is configured to *not* fsync() on all Append() calls
I20260620 23:36:00.329425 4044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:00.305738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40440 (local address 127.0.97.62:43533)
0620 23:36:00.306519 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:00.306537 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:00.306593 (+ 56us) server_negotiation.cc:407] Connection header received
0620 23:36:00.308900 (+ 2307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:00.308929 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:00.309425 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:00.309896 (+ 471us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:00.311243 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:00.313353 (+ 2110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:00.314698 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:00.315738 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:00.315937 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:00.319997 (+ 4060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:00.320009 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:00.320030 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:00.320081 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:00.324488 (+ 4407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:00.325299 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:00.325309 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:00.325318 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:00.325543 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:00.325905 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:00.325915 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:00.325921 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:00.326684 (+ 763us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:00.327066 (+ 382us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:00.327486 (+ 420us) server_negotiation.cc:299] Negotiation successful
0620 23:36:00.327964 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":325,"threads_started":1}
I20260620 23:36:00.329962 4041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9155caa57ec84308a655a111a9b03c2f: No bootstrap required, opened a new log
I20260620 23:36:00.338474 4041 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9155caa57ec84308a655a111a9b03c2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9155caa57ec84308a655a111a9b03c2f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43533 } }
I20260620 23:36:00.339033 4041 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9155caa57ec84308a655a111a9b03c2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:36:00.339195 4041 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9155caa57ec84308a655a111a9b03c2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9155caa57ec84308a655a111a9b03c2f, State: Initialized, Role: FOLLOWER
I20260620 23:36:00.339844 4041 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9155caa57ec84308a655a111a9b03c2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9155caa57ec84308a655a111a9b03c2f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43533 } }
I20260620 23:36:00.340098 4041 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9155caa57ec84308a655a111a9b03c2f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:36:00.340253 4041 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9155caa57ec84308a655a111a9b03c2f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:36:00.340438 4041 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9155caa57ec84308a655a111a9b03c2f [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:36:00.342607 4041 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9155caa57ec84308a655a111a9b03c2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9155caa57ec84308a655a111a9b03c2f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43533 } }
I20260620 23:36:00.343286 4041 leader_election.cc:302] T 00000000000000000000000000000000 P 9155caa57ec84308a655a111a9b03c2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9155caa57ec84308a655a111a9b03c2f; no voters:
I20260620 23:36:00.344096 4041 leader_election.cc:288] T 00000000000000000000000000000000 P 9155caa57ec84308a655a111a9b03c2f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:36:00.344259 4046 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9155caa57ec84308a655a111a9b03c2f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:36:00.345091 4046 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9155caa57ec84308a655a111a9b03c2f [term 1 LEADER]: Becoming Leader. State: Replica: 9155caa57ec84308a655a111a9b03c2f, State: Running, Role: LEADER
I20260620 23:36:00.345754 4046 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9155caa57ec84308a655a111a9b03c2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9155caa57ec84308a655a111a9b03c2f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43533 } }
I20260620 23:36:00.346534 4041 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9155caa57ec84308a655a111a9b03c2f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:36:00.348839 4048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9155caa57ec84308a655a111a9b03c2f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9155caa57ec84308a655a111a9b03c2f. Latest consensus state: current_term: 1 leader_uuid: "9155caa57ec84308a655a111a9b03c2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9155caa57ec84308a655a111a9b03c2f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43533 } } }
I20260620 23:36:00.349283 4048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9155caa57ec84308a655a111a9b03c2f [sys.catalog]: This master's current role is: LEADER
I20260620 23:36:00.349817 4047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9155caa57ec84308a655a111a9b03c2f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9155caa57ec84308a655a111a9b03c2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9155caa57ec84308a655a111a9b03c2f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43533 } } }
I20260620 23:36:00.350214 4047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9155caa57ec84308a655a111a9b03c2f [sys.catalog]: This master's current role is: LEADER
W20260620 23:36:00.359035 4062 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9155caa57ec84308a655a111a9b03c2f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:36:00.359157 4062 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:36:00.359301 4053 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:36:00.361529 4053 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:36:00.371219 4053 catalog_manager.cc:1347] Generated new cluster ID: 933f26e4dca8430ca7131b6e09eb4d65
I20260620 23:36:00.371349 4053 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:36:00.383895 4053 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:36:00.385293 4053 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:36:00.395527 4053 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9155caa57ec84308a655a111a9b03c2f: Generated new TSK 0
I20260620 23:36:00.396472 4053 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 23:36:00.455605 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43533
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:36:00.796448 4067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:36:00.796733 4067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:36:00.796921 4067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:36:00.804258 4067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:36:00.804378 4067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:36:00.804553 4067 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 23:36:00.814241 4067 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43533
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:36:00.815670 4067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:36:00.817291 4067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:36:00.831916 4073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:00.831962 4072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:00.832158 4075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:36:00.833478 4067 server_base.cc:1034] running on GCE node
I20260620 23:36:00.834239 4067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:36:00.835407 4067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:36:00.836678 4067 hybrid_clock.cc:648] HybridClock initialized: now 1781998560836634 us; error 48 us; skew 500 ppm
I20260620 23:36:00.837011 4067 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:36:00 dist-test-slave-tcgx krb5kdc[3961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998560, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:36:00.840396 4067 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 23:36:00.842020 4067 webserver.cc:460] Webserver started at http://127.0.97.1:40995/ using document root <none> and password file <none>
I20260620 23:36:00.842775 4067 fs_manager.cc:359] Metadata directory not provided
I20260620 23:36:00.842974 4067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:36:00.843318 4067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:36:00.846306 4067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/minicluster-data/ts-0/data/instance:
uuid: "112d67005c314c4ab62b36b0433e0ee1"
format_stamp: "Formatted at 2026-06-20 23:36:00 on dist-test-slave-tcgx"
server_key: "cdb80b44ccdcb35de6ddd298ace24ec8"
server_key_iv: "70e5867593e523f98f2c204d79826a14"
server_key_version: "encryptionkey@0"
I20260620 23:36:00.847268 4067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/minicluster-data/ts-0/wal/instance:
uuid: "112d67005c314c4ab62b36b0433e0ee1"
format_stamp: "Formatted at 2026-06-20 23:36:00 on dist-test-slave-tcgx"
server_key: "cdb80b44ccdcb35de6ddd298ace24ec8"
server_key_iv: "70e5867593e523f98f2c204d79826a14"
server_key_version: "encryptionkey@0"
I20260620 23:36:00.852742 4067 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 23:36:00.856774 4082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:00.858192 4067 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 23:36:00.858445 4067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/minicluster-data/ts-0/wal
uuid: "112d67005c314c4ab62b36b0433e0ee1"
format_stamp: "Formatted at 2026-06-20 23:36:00 on dist-test-slave-tcgx"
server_key: "cdb80b44ccdcb35de6ddd298ace24ec8"
server_key_iv: "70e5867593e523f98f2c204d79826a14"
server_key_version: "encryptionkey@0"
I20260620 23:36:00.858670 4067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:36:00.876648 4067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:36:00.881014 4067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:36:00.881318 4067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:36:00.882268 4067 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:36:00.883656 4067 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:36:00.883759 4067 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:00.883913 4067 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:36:00.883986 4067 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:00.917575 4067 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37969
I20260620 23:36:00.917640 4194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37969 every 8 connection(s)
I20260620 23:36:00.918962 4067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
I20260620 23:36:00.928208 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 4067
I20260620 23:36:00.928447 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/minicluster-data/ts-0/wal/instance
I20260620 23:36:00.929061 388 external_mini_cluster.cc:1428] Setting key e792216ee6f69977ccf7f8b286c864e2
Jun 20 23:36:00 dist-test-slave-tcgx krb5kdc[3961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998560, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:36:00.938807 4197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:00.921566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37437 (local address 127.0.97.62:43533)
0620 23:36:00.922015 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:00.922020 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:00.923106 (+ 1086us) server_negotiation.cc:407] Connection header received
0620 23:36:00.924937 (+ 1831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:00.924943 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:00.925036 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:00.925152 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:00.926656 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:00.927460 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:00.929163 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:00.929560 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:00.929695 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:00.933219 (+ 3524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:00.933224 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:00.933227 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:00.933260 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:00.935811 (+ 2551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:00.936545 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:00.936553 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:00.936558 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:00.936677 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:00.937097 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:00.937103 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:00.937107 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:00.937367 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:00.937509 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:00.938223 (+ 714us) server_negotiation.cc:299] Negotiation successful
0620 23:36:00.938433 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":220,"threads_started":1}
I20260620 23:36:00.939839 4198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:00.922180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43533 (local address 127.0.97.1:37437)
0620 23:36:00.922893 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:00.922969 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:00.924650 (+ 1681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:00.925402 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:00.925411 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:00.925938 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:00.926482 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:00.926499 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:00.927623 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:00.927630 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:00.929015 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:00.929027 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:00.929798 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:00.929805 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:00.930030 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:00.930757 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:00.930801 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:00.932999 (+ 2198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:36:00.935959 (+ 2960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:00.935968 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:00.935987 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:00.936424 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:00.936790 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:00.936796 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:00.936800 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:00.936986 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:00.937501 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:36:00.937509 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:36:00.937868 (+ 359us) client_negotiation.cc:769] Sending connection context
0620 23:36:00.938139 (+ 271us) client_negotiation.cc:240] Negotiation successful
0620 23:36:00.938691 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":288,"threads_started":1}
I20260620 23:36:00.941440 4195 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43533
I20260620 23:36:00.941771 4195 heartbeater.cc:464] Registering TS with master...
I20260620 23:36:00.942694 4195 heartbeater.cc:511] Master 127.0.97.62:43533 requested a full tablet report, sending...
I20260620 23:36:00.945334 4006 ts_manager.cc:194] Registered new tserver with Master: 112d67005c314c4ab62b36b0433e0ee1 (127.0.97.1:37969)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 23:36:00.947602 4006 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:37437
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 23:36:00.974830 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43533
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:36:01.310617 4201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:36:01.311053 4201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:36:01.311208 4201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:36:01.323098 4201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:36:01.323264 4201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:36:01.323449 4201 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 23:36:01.335793 4201 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43533
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:36:01.337277 4201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:36:01.339036 4201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:36:01.355916 4206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:01.356251 4207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:01.356998 4209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:36:01.358281 4201 server_base.cc:1034] running on GCE node
I20260620 23:36:01.359234 4201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:36:01.360612 4201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:36:01.361871 4201 hybrid_clock.cc:648] HybridClock initialized: now 1781998561361811 us; error 52 us; skew 500 ppm
I20260620 23:36:01.362365 4201 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:36:01 dist-test-slave-tcgx krb5kdc[3961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:36:01.366782 4201 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 23:36:01.368562 4201 webserver.cc:460] Webserver started at http://127.0.97.2:36695/ using document root <none> and password file <none>
I20260620 23:36:01.369325 4201 fs_manager.cc:359] Metadata directory not provided
I20260620 23:36:01.369499 4201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:36:01.369825 4201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:36:01.372871 4201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/minicluster-data/ts-1/data/instance:
uuid: "7b50ee224d0043d1ae8d14e437a24d87"
format_stamp: "Formatted at 2026-06-20 23:36:01 on dist-test-slave-tcgx"
server_key: "6dd8a0f8c9bb770c108cf8da0107295f"
server_key_iv: "adfb75251d4147364c0be548254c6c3c"
server_key_version: "encryptionkey@0"
I20260620 23:36:01.373670 4201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/minicluster-data/ts-1/wal/instance:
uuid: "7b50ee224d0043d1ae8d14e437a24d87"
format_stamp: "Formatted at 2026-06-20 23:36:01 on dist-test-slave-tcgx"
server_key: "6dd8a0f8c9bb770c108cf8da0107295f"
server_key_iv: "adfb75251d4147364c0be548254c6c3c"
server_key_version: "encryptionkey@0"
I20260620 23:36:01.379366 4201 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 23:36:01.383342 4216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:01.385012 4201 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 23:36:01.385206 4201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/minicluster-data/ts-1/wal
uuid: "7b50ee224d0043d1ae8d14e437a24d87"
format_stamp: "Formatted at 2026-06-20 23:36:01 on dist-test-slave-tcgx"
server_key: "6dd8a0f8c9bb770c108cf8da0107295f"
server_key_iv: "adfb75251d4147364c0be548254c6c3c"
server_key_version: "encryptionkey@0"
I20260620 23:36:01.385434 4201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:36:01.412246 4201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:36:01.417196 4201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:36:01.417521 4201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:36:01.418653 4201 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:36:01.420316 4201 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:36:01.420423 4201 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:01.420578 4201 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:36:01.420639 4201 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:01.459522 4201 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34645
I20260620 23:36:01.459556 4328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34645 every 8 connection(s)
I20260620 23:36:01.460858 4201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
I20260620 23:36:01.466636 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 4201
I20260620 23:36:01.466836 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/minicluster-data/ts-1/wal/instance
I20260620 23:36:01.467324 388 external_mini_cluster.cc:1428] Setting key 47f28ad2e3915d263aa6d2f02b2d0375
Jun 20 23:36:01 dist-test-slave-tcgx krb5kdc[3961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:36:01.486140 4331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:01.463973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50255 (local address 127.0.97.62:43533)
0620 23:36:01.464490 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:01.464496 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:01.465785 (+ 1289us) server_negotiation.cc:407] Connection header received
0620 23:36:01.469274 (+ 3489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:01.469282 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:01.469379 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:01.469496 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:01.471894 (+ 2398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:01.473186 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:01.475597 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:01.476024 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:01.476172 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:01.479923 (+ 3751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:01.479928 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:01.479931 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:01.479964 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:01.482645 (+ 2681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:01.483541 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:01.483549 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:01.483557 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:01.483676 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:01.484145 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:01.484156 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:01.484159 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:01.484449 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:01.484612 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:01.485511 (+ 899us) server_negotiation.cc:299] Negotiation successful
0620 23:36:01.485771 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":261,"threads_started":1}
I20260620 23:36:01.487419 4332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:01.464622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43533 (local address 127.0.97.2:50255)
0620 23:36:01.465565 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:01.465623 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:01.468764 (+ 3141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:01.469827 (+ 1063us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:01.469842 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:01.470709 (+ 867us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:01.471667 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:01.471699 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:01.473391 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:01.473400 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:01.475420 (+ 2020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:01.475438 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:01.476266 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:01.476273 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:01.476521 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:01.477245 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:01.477270 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:01.479654 (+ 2384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:36:01.482839 (+ 3185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:01.482849 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:01.482869 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:01.483402 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:01.483807 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:01.483813 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:01.483817 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:01.484019 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:01.484587 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:36:01.484597 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:36:01.485009 (+ 412us) client_negotiation.cc:769] Sending connection context
0620 23:36:01.485411 (+ 402us) client_negotiation.cc:240] Negotiation successful
0620 23:36:01.485998 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":338,"threads_started":1}
I20260620 23:36:01.489089 4329 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43533
I20260620 23:36:01.489488 4329 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 23:36:01.490509 4329 heartbeater.cc:511] Master 127.0.97.62:43533 requested a full tablet report, sending...
I20260620 23:36:01.492177 4006 ts_manager.cc:194] Registered new tserver with Master: 7b50ee224d0043d1ae8d14e437a24d87 (127.0.97.2:34645)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260620 23:36:01.493372 4006 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:50255
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 23:36:01.526279 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43533
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:36:01.875341 4335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:36:01.875635 4335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:36:01.875735 4335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:36:01.883131 4335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:36:01.883255 4335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:36:01.883370 4335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 23:36:01.893520 4335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43533
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:36:01.894819 4335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:36:01.896164 4335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:36:01.909648 4340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:01.909577 4343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:01.909621 4341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:36:01.910678 4335 server_base.cc:1034] running on GCE node
I20260620 23:36:01.911384 4335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:36:01.912348 4335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:36:01.913610 4335 hybrid_clock.cc:648] HybridClock initialized: now 1781998561913546 us; error 55 us; skew 500 ppm
I20260620 23:36:01.913993 4335 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:36:01 dist-test-slave-tcgx krb5kdc[3961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:36:01.917279 4335 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 23:36:01.920423 4335 webserver.cc:460] Webserver started at http://127.0.97.3:40261/ using document root <none> and password file <none>
I20260620 23:36:01.921129 4335 fs_manager.cc:359] Metadata directory not provided
I20260620 23:36:01.921273 4335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:36:01.921589 4335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:36:01.924575 4335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/minicluster-data/ts-2/data/instance:
uuid: "251fead6dc014dbd909fa7fdf8c95f68"
format_stamp: "Formatted at 2026-06-20 23:36:01 on dist-test-slave-tcgx"
server_key: "c0a500a357f71d34dead67441b3fc110"
server_key_iv: "e90ef932e0faeda70c44af7dacf1fbee"
server_key_version: "encryptionkey@0"
I20260620 23:36:01.925380 4335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/minicluster-data/ts-2/wal/instance:
uuid: "251fead6dc014dbd909fa7fdf8c95f68"
format_stamp: "Formatted at 2026-06-20 23:36:01 on dist-test-slave-tcgx"
server_key: "c0a500a357f71d34dead67441b3fc110"
server_key_iv: "e90ef932e0faeda70c44af7dacf1fbee"
server_key_version: "encryptionkey@0"
I20260620 23:36:01.930594 4335 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 23:36:01.934338 4350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:01.935829 4335 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 23:36:01.936014 4335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/minicluster-data/ts-2/wal
uuid: "251fead6dc014dbd909fa7fdf8c95f68"
format_stamp: "Formatted at 2026-06-20 23:36:01 on dist-test-slave-tcgx"
server_key: "c0a500a357f71d34dead67441b3fc110"
server_key_iv: "e90ef932e0faeda70c44af7dacf1fbee"
server_key_version: "encryptionkey@0"
I20260620 23:36:01.936210 4335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:36:01.950698 4195 heartbeater.cc:503] Master 127.0.97.62:43533 was elected leader, sending a full tablet report...
I20260620 23:36:01.955793 4335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:36:01.960173 4335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:36:01.960467 4335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:36:01.961498 4335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:36:01.963116 4335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:36:01.963213 4335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:01.963367 4335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:36:01.963439 4335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:01.997982 4335 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36599
I20260620 23:36:01.998023 4462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36599 every 8 connection(s)
I20260620 23:36:01.999440 4335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
I20260620 23:36:02.009506 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 4335
I20260620 23:36:02.009646 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998525501163-388-0/minicluster-data/ts-2/wal/instance
I20260620 23:36:02.010233 388 external_mini_cluster.cc:1428] Setting key ea8f2a897ddd371ef4874d6e3115eb3a
Jun 20 23:36:02 dist-test-slave-tcgx krb5kdc[3961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:36:02.019855 4465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:02.002036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57541 (local address 127.0.97.62:43533)
0620 23:36:02.002488 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:02.002495 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:02.003569 (+ 1074us) server_negotiation.cc:407] Connection header received
0620 23:36:02.005517 (+ 1948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:02.005521 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:02.005591 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:02.005706 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:02.007420 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:02.008182 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:02.009888 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:02.010498 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:02.010644 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:02.014302 (+ 3658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:02.014308 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:02.014313 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:02.014351 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:02.016796 (+ 2445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:02.017526 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:02.017532 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:02.017536 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:02.017659 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:02.018080 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:02.018085 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:02.018088 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:02.018350 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:02.018493 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:02.019245 (+ 752us) server_negotiation.cc:299] Negotiation successful
0620 23:36:02.019468 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":211,"threads_started":1}
I20260620 23:36:02.020789 4466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:02.002575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43533 (local address 127.0.97.3:57541)
0620 23:36:02.003393 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:02.003442 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:02.005226 (+ 1784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:02.005944 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:02.005954 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:02.006525 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:02.007243 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:02.007265 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:02.008349 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:02.008353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:02.009728 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:02.009746 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:02.010712 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:02.010717 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:02.011048 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:02.011811 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:02.011839 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:02.014095 (+ 2256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:36:02.016938 (+ 2843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:02.016948 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:02.016968 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:02.017401 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:02.017762 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:02.017769 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:02.017773 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:02.017963 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:02.018471 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:36:02.018480 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:36:02.018830 (+ 350us) client_negotiation.cc:769] Sending connection context
0620 23:36:02.019145 (+ 315us) client_negotiation.cc:240] Negotiation successful
0620 23:36:02.019687 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":357,"threads_started":1}
I20260620 23:36:02.022161 4463 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43533
I20260620 23:36:02.022476 4463 heartbeater.cc:464] Registering TS with master...
I20260620 23:36:02.023272 4463 heartbeater.cc:511] Master 127.0.97.62:43533 requested a full tablet report, sending...
I20260620 23:36:02.024811 4006 ts_manager.cc:194] Registered new tserver with Master: 251fead6dc014dbd909fa7fdf8c95f68 (127.0.97.3:36599)
I20260620 23:36:02.025859 4006 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:57541
I20260620 23:36:02.025907 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 23:36:02 dist-test-slave-tcgx krb5kdc[3961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 23:36:02 dist-test-slave-tcgx krb5kdc[3961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 23:36:02.075881 4471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:02.060299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38200 (local address 127.0.97.1:37969)
0620 23:36:02.060881 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:02.060889 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:02.060938 (+ 49us) server_negotiation.cc:407] Connection header received
0620 23:36:02.061025 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:02.061032 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:02.061256 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:02.061386 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:02.062922 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:02.064130 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:02.065335 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:02.065909 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:02.066006 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:02.069996 (+ 3990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:02.070009 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:02.070025 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:02.070062 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:02.073108 (+ 3046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:02.073832 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:02.073839 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:02.073841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:02.073976 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:02.074338 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:02.074341 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:02.074343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:02.074672 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:02.074907 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:02.075251 (+ 344us) server_negotiation.cc:299] Negotiation successful
0620 23:36:02.075454 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":261,"threads_started":1}
Jun 20 23:36:02 dist-test-slave-tcgx krb5kdc[3961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:36:02.094332 4465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:02.080317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40456 (local address 127.0.97.62:43533)
0620 23:36:02.080538 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:02.080545 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:02.080707 (+ 162us) server_negotiation.cc:407] Connection header received
0620 23:36:02.080916 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:02.080921 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:02.081000 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:02.081103 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:02.082356 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:02.083184 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:02.084645 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:02.085056 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:02.085193 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:02.089297 (+ 4104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:02.089302 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:02.089306 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:02.089342 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:02.091860 (+ 2518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:02.092495 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:02.092499 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:02.092501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:02.092674 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:02.093022 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:02.093026 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:02.093028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:02.093354 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:02.093539 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:02.093806 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 23:36:02.093980 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260620 23:36:02.094990 4006 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40456
I20260620 23:36:02.096155 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 4067
I20260620 23:36:02.109998 4471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:02.097074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38208 (local address 127.0.97.1:37969)
0620 23:36:02.097333 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:02.097338 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:02.097554 (+ 216us) server_negotiation.cc:407] Connection header received
0620 23:36:02.097689 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:02.097693 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:02.097768 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:02.097848 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:02.099232 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:02.099968 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:02.101265 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:02.101674 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:02.101775 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:02.104308 (+ 2533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:02.104313 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:02.104317 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:02.104360 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:02.107423 (+ 3063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:02.108149 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:02.108153 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:02.108156 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:02.108284 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:02.108681 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:02.108684 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:02.108686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:02.108936 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:02.109111 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:02.109428 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 23:36:02.109598 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260620 23:36:02.110688 4190 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:36:02.193560 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 4067
I20260620 23:36:02.210479 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 4201
Jun 20 23:36:02 dist-test-slave-tcgx krb5kdc[3961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 23:36:02.228067 4476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:02.211501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52394 (local address 127.0.97.2:34645)
0620 23:36:02.212080 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:02.212088 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:02.212115 (+ 27us) server_negotiation.cc:407] Connection header received
0620 23:36:02.212177 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:02.212183 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:02.212421 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:02.212550 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:02.213788 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:02.214811 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:02.216678 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:02.217301 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:02.217426 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:02.222168 (+ 4742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:02.222183 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:02.222198 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:02.222238 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:02.225142 (+ 2904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:02.225865 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:02.225873 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:02.225875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:02.225998 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:02.226410 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:02.226415 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:02.226418 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:02.226783 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:02.227130 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:02.227440 (+ 310us) server_negotiation.cc:299] Negotiation successful
0620 23:36:02.227701 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":297,"threads_started":1}
I20260620 23:36:02.229353 4324 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:36:02.312271 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 4201
I20260620 23:36:02.328521 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 4335
Jun 20 23:36:02 dist-test-slave-tcgx krb5kdc[3961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 23:36:02.345069 4478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:02.329533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40860 (local address 127.0.97.3:36599)
0620 23:36:02.330187 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:02.330195 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:02.330225 (+ 30us) server_negotiation.cc:407] Connection header received
0620 23:36:02.330289 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:02.330295 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:02.330542 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:02.330677 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:02.332007 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:02.333005 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:02.334310 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:02.334880 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:02.335075 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:02.339328 (+ 4253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:02.339341 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:02.339356 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:02.339411 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:02.342541 (+ 3130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:02.343287 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:02.343295 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:02.343298 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:02.343443 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:02.343750 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:02.343753 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:02.343755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:02.344083 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:02.344322 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:02.344592 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 23:36:02.344783 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":356,"threads_started":1}
I20260620 23:36:02.346393 4458 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:36:02.430786 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 4335
I20260620 23:36:02.447727 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 3974
I20260620 23:36:02.448992 4036 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:36:02.553817 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 3974
2026-06-20T23:36:02Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3996 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:36:02 dist-test-slave-tcgx krb5kdc[4483](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:36:02 dist-test-slave-tcgx krb5kdc[4483](info): set up 2 sockets
Jun 20 23:36:02 dist-test-slave-tcgx krb5kdc[4483](info): commencing operation
krb5kdc: starting...
W20260620 23:36:03.655498 388 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/krb5kdc/test-user.keytab.
Jun 20 23:36:03 dist-test-slave-tcgx krb5kdc[4483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:36:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:36:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 23:36:03.856148 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34805
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34805
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:36:04.202244 4496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:36:04.202553 4496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:36:04.202741 4496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:36:04.210229 4496 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:36:04.210361 4496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:36:04.210474 4496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:36:04.210583 4496 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:36:04.210655 4496 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:36:04.220892 4496 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34805
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:36:04.222308 4496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:36:04.223698 4496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:36:04.235738 4501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:04.235745 4504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:04.235971 4502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:36:04.236716 4496 server_base.cc:1034] running on GCE node
I20260620 23:36:04.237529 4496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:36:04.239105 4496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:36:04.240325 4496 hybrid_clock.cc:648] HybridClock initialized: now 1781998564240256 us; error 69 us; skew 500 ppm
I20260620 23:36:04.240672 4496 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:36:04 dist-test-slave-tcgx krb5kdc[4483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:36:04.243831 4496 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 23:36:04.245024 4496 fs_manager.cc:359] Metadata directory not provided
I20260620 23:36:04.245162 4496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:36:04.245487 4496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:36:04.248548 4496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/minicluster-data/master-0/data/instance:
uuid: "a5d93bbd3950430a824ef421660b46ac"
format_stamp: "Formatted at 2026-06-20 23:36:04 on dist-test-slave-tcgx"
server_key: "06e6123e3874b7fb2f566c83b84df26e"
server_key_iv: "6c8e97b6225e008f7b2fe1c8debc08eb"
server_key_version: "encryptionkey@0"
I20260620 23:36:04.249357 4496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/minicluster-data/master-0/wal/instance:
uuid: "a5d93bbd3950430a824ef421660b46ac"
format_stamp: "Formatted at 2026-06-20 23:36:04 on dist-test-slave-tcgx"
server_key: "06e6123e3874b7fb2f566c83b84df26e"
server_key_iv: "6c8e97b6225e008f7b2fe1c8debc08eb"
server_key_version: "encryptionkey@0"
I20260620 23:36:04.254494 4496 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 23:36:04.258073 4510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:04.259593 4496 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 23:36:04.259805 4496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/minicluster-data/master-0/wal
uuid: "a5d93bbd3950430a824ef421660b46ac"
format_stamp: "Formatted at 2026-06-20 23:36:04 on dist-test-slave-tcgx"
server_key: "06e6123e3874b7fb2f566c83b84df26e"
server_key_iv: "6c8e97b6225e008f7b2fe1c8debc08eb"
server_key_version: "encryptionkey@0"
I20260620 23:36:04.260051 4496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:36:04.275276 4496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:36:04.279986 4496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:36:04.280284 4496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:36:04.298743 4496 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34805
I20260620 23:36:04.298719 4561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34805 every 8 connection(s)
I20260620 23:36:04.299724 4496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:36:04.303465 4562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:36:04.306013 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 4496
I20260620 23:36:04.306233 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/minicluster-data/master-0/wal/instance
I20260620 23:36:04.306869 388 external_mini_cluster.cc:1428] Setting key 2ccc3814125e9dd1057c46a99267d844
I20260620 23:36:04.316128 4562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5d93bbd3950430a824ef421660b46ac: Bootstrap starting.
I20260620 23:36:04.320156 4562 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5d93bbd3950430a824ef421660b46ac: Neither blocks nor log segments found. Creating new log.
I20260620 23:36:04.321565 4562 log.cc:824] T 00000000000000000000000000000000 P a5d93bbd3950430a824ef421660b46ac: Log is configured to *not* fsync() on all Append() calls
Jun 20 23:36:04 dist-test-slave-tcgx krb5kdc[4483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:36:04.324621 4562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5d93bbd3950430a824ef421660b46ac: No bootstrap required, opened a new log
I20260620 23:36:04.331065 4565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:04.309066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39364 (local address 127.0.97.62:34805)
0620 23:36:04.309869 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:04.309882 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:04.309927 (+ 45us) server_negotiation.cc:407] Connection header received
0620 23:36:04.311692 (+ 1765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:04.311717 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:04.312168 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:04.312537 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:04.314446 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:04.315827 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:04.317628 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:04.318337 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:04.318481 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:04.323436 (+ 4955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:04.323444 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:04.323478 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:04.323514 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:04.326646 (+ 3132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:04.327504 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:04.327511 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:04.327518 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:04.327678 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:04.328096 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:04.328101 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:04.328105 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:04.328656 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:04.329001 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:04.329517 (+ 516us) server_negotiation.cc:299] Negotiation successful
0620 23:36:04.329929 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":370,"threads_started":1}
I20260620 23:36:04.332720 4562 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5d93bbd3950430a824ef421660b46ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5d93bbd3950430a824ef421660b46ac" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34805 } }
I20260620 23:36:04.333194 4562 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5d93bbd3950430a824ef421660b46ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:36:04.333316 4562 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5d93bbd3950430a824ef421660b46ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5d93bbd3950430a824ef421660b46ac, State: Initialized, Role: FOLLOWER
I20260620 23:36:04.333982 4562 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5d93bbd3950430a824ef421660b46ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5d93bbd3950430a824ef421660b46ac" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34805 } }
I20260620 23:36:04.334247 4562 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5d93bbd3950430a824ef421660b46ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:36:04.334384 4562 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5d93bbd3950430a824ef421660b46ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:36:04.334508 4562 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5d93bbd3950430a824ef421660b46ac [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:36:04.336699 4562 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5d93bbd3950430a824ef421660b46ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5d93bbd3950430a824ef421660b46ac" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34805 } }
I20260620 23:36:04.337312 4562 leader_election.cc:302] T 00000000000000000000000000000000 P a5d93bbd3950430a824ef421660b46ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5d93bbd3950430a824ef421660b46ac; no voters:
I20260620 23:36:04.337936 4562 leader_election.cc:288] T 00000000000000000000000000000000 P a5d93bbd3950430a824ef421660b46ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:36:04.338097 4567 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5d93bbd3950430a824ef421660b46ac [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:36:04.339267 4567 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5d93bbd3950430a824ef421660b46ac [term 1 LEADER]: Becoming Leader. State: Replica: a5d93bbd3950430a824ef421660b46ac, State: Running, Role: LEADER
I20260620 23:36:04.339970 4567 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5d93bbd3950430a824ef421660b46ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5d93bbd3950430a824ef421660b46ac" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34805 } }
I20260620 23:36:04.340330 4562 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5d93bbd3950430a824ef421660b46ac [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:36:04.343032 4568 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5d93bbd3950430a824ef421660b46ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a5d93bbd3950430a824ef421660b46ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5d93bbd3950430a824ef421660b46ac" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34805 } } }
I20260620 23:36:04.343489 4568 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5d93bbd3950430a824ef421660b46ac [sys.catalog]: This master's current role is: LEADER
I20260620 23:36:04.343091 4569 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5d93bbd3950430a824ef421660b46ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5d93bbd3950430a824ef421660b46ac. Latest consensus state: current_term: 1 leader_uuid: "a5d93bbd3950430a824ef421660b46ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5d93bbd3950430a824ef421660b46ac" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34805 } } }
I20260620 23:36:04.344254 4569 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5d93bbd3950430a824ef421660b46ac [sys.catalog]: This master's current role is: LEADER
I20260620 23:36:04.345817 4576 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:36:04.351539 4576 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:36:04.360414 4576 catalog_manager.cc:1347] Generated new cluster ID: 8f23d5c0fce64215b10f2ad0a3e52409
I20260620 23:36:04.360589 4576 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:36:04.380882 4576 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:36:04.382328 4576 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:36:04.391506 4576 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a5d93bbd3950430a824ef421660b46ac: Generated new TSK 0
I20260620 23:36:04.392441 4576 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 23:36:04.447481 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34805
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:36:04.787544 4588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:36:04.787835 4588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:36:04.787935 4588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:36:04.795233 4588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:36:04.795414 4588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:36:04.795576 4588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 23:36:04.805570 4588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34805
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:36:04.806998 4588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:36:04.808300 4588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:36:04.821549 4593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:04.821640 4596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:04.821593 4594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:36:04.827601 4588 server_base.cc:1034] running on GCE node
I20260620 23:36:04.828243 4588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:36:04.829279 4588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:36:04.830488 4588 hybrid_clock.cc:648] HybridClock initialized: now 1781998564830420 us; error 62 us; skew 500 ppm
I20260620 23:36:04.830861 4588 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:36:04 dist-test-slave-tcgx krb5kdc[4483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:36:04.834272 4588 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 23:36:04.835629 4588 fs_manager.cc:359] Metadata directory not provided
I20260620 23:36:04.835752 4588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:36:04.836112 4588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:36:04.839315 4588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/minicluster-data/ts-0/data/instance:
uuid: "56d963cb3a8544a88c29e0f233172e98"
format_stamp: "Formatted at 2026-06-20 23:36:04 on dist-test-slave-tcgx"
server_key: "883bd2cd2453a62844a31d7b7043dccf"
server_key_iv: "b1ee8c3f0af6b47d9bfc623c1dc943e4"
server_key_version: "encryptionkey@0"
I20260620 23:36:04.840209 4588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/minicluster-data/ts-0/wal/instance:
uuid: "56d963cb3a8544a88c29e0f233172e98"
format_stamp: "Formatted at 2026-06-20 23:36:04 on dist-test-slave-tcgx"
server_key: "883bd2cd2453a62844a31d7b7043dccf"
server_key_iv: "b1ee8c3f0af6b47d9bfc623c1dc943e4"
server_key_version: "encryptionkey@0"
I20260620 23:36:04.845734 4588 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:36:04.849315 4602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:04.850874 4588 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260620 23:36:04.851143 4588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/minicluster-data/ts-0/wal
uuid: "56d963cb3a8544a88c29e0f233172e98"
format_stamp: "Formatted at 2026-06-20 23:36:04 on dist-test-slave-tcgx"
server_key: "883bd2cd2453a62844a31d7b7043dccf"
server_key_iv: "b1ee8c3f0af6b47d9bfc623c1dc943e4"
server_key_version: "encryptionkey@0"
I20260620 23:36:04.851394 4588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:36:04.881075 4588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:36:04.885895 4588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:36:04.886255 4588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:36:04.887414 4588 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:36:04.888893 4588 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:36:04.888988 4588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:04.889071 4588 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:36:04.889123 4588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:04.925197 4588 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42509
I20260620 23:36:04.925243 4714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42509 every 8 connection(s)
I20260620 23:36:04.926196 4588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
I20260620 23:36:04.928206 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 4588
I20260620 23:36:04.928406 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/minicluster-data/ts-0/wal/instance
I20260620 23:36:04.929313 388 external_mini_cluster.cc:1428] Setting key a211f8e70e798c026e8937515a69f6e5
Jun 20 23:36:04 dist-test-slave-tcgx krb5kdc[4483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 23:36:04.950109 4717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:04.928799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60807 (local address 127.0.97.62:34805)
0620 23:36:04.929291 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:04.929297 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:04.930482 (+ 1185us) server_negotiation.cc:407] Connection header received
0620 23:36:04.932437 (+ 1955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:04.932446 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:04.932538 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:04.932662 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:04.934213 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:04.935453 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:04.937612 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:04.938054 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:04.938172 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:04.943292 (+ 5120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:04.943299 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:04.943302 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:04.943339 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:04.946406 (+ 3067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:04.947473 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:04.947482 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:04.947486 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:04.947626 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:04.948139 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:04.948144 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:04.948146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:04.948440 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:04.948612 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:04.949441 (+ 829us) server_negotiation.cc:299] Negotiation successful
0620 23:36:04.949683 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":242,"threads_started":1}
I20260620 23:36:04.951531 4718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:04.929485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34805 (local address 127.0.97.1:60807)
0620 23:36:04.930322 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:04.930370 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:04.932149 (+ 1779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:04.932865 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:04.932874 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:04.933426 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:04.934016 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:04.934034 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:04.935612 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:04.935616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:04.937401 (+ 1785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:04.937421 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:04.938359 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:04.938371 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:04.938752 (+ 381us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:04.939685 (+ 933us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:04.939731 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:04.942992 (+ 3261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:36:04.946617 (+ 3625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:04.946629 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:04.946651 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:04.947318 (+ 667us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:04.947757 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:04.947763 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:04.947767 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:04.947991 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:04.948588 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:36:04.948597 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:36:04.949014 (+ 417us) client_negotiation.cc:769] Sending connection context
0620 23:36:04.949332 (+ 318us) client_negotiation.cc:240] Negotiation successful
0620 23:36:04.949989 (+ 657us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":380,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260620 23:36:04.953356 4715 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34805
I20260620 23:36:04.953704 4715 heartbeater.cc:464] Registering TS with master...
I20260620 23:36:04.954407 4715 heartbeater.cc:511] Master 127.0.97.62:34805 requested a full tablet report, sending...
I20260620 23:36:04.957019 4527 ts_manager.cc:194] Registered new tserver with Master: 56d963cb3a8544a88c29e0f233172e98 (127.0.97.1:42509)
I20260620 23:36:04.958750 4527 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:60807
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 23:36:04.979177 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34805
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:36:05.324692 4721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:36:05.325016 4721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:36:05.325176 4721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:36:05.332810 4721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:36:05.332943 4721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:36:05.333130 4721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 23:36:05.343551 4721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34805
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:36:05.345039 4721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:36:05.346411 4721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:36:05.360019 4726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:05.360008 4729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:05.360147 4727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:36:05.361582 4721 server_base.cc:1034] running on GCE node
I20260620 23:36:05.362323 4721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:36:05.363409 4721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:36:05.364676 4721 hybrid_clock.cc:648] HybridClock initialized: now 1781998565364627 us; error 43 us; skew 500 ppm
I20260620 23:36:05.365065 4721 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:36:05 dist-test-slave-tcgx krb5kdc[4483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998565, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:36:05.372648 4721 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 23:36:05.373940 4721 fs_manager.cc:359] Metadata directory not provided
I20260620 23:36:05.374099 4721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:36:05.374459 4721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:36:05.377640 4721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/minicluster-data/ts-1/data/instance:
uuid: "64240eb20c9a4d38862ac49b1f77ec30"
format_stamp: "Formatted at 2026-06-20 23:36:05 on dist-test-slave-tcgx"
server_key: "f01109ecb5c5f41d077c29a5b2141458"
server_key_iv: "f62de12dcbff465eadf9f5a18800be5d"
server_key_version: "encryptionkey@0"
I20260620 23:36:05.378518 4721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/minicluster-data/ts-1/wal/instance:
uuid: "64240eb20c9a4d38862ac49b1f77ec30"
format_stamp: "Formatted at 2026-06-20 23:36:05 on dist-test-slave-tcgx"
server_key: "f01109ecb5c5f41d077c29a5b2141458"
server_key_iv: "f62de12dcbff465eadf9f5a18800be5d"
server_key_version: "encryptionkey@0"
I20260620 23:36:05.384078 4721 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260620 23:36:05.387982 4735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:05.389418 4721 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260620 23:36:05.389611 4721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/minicluster-data/ts-1/wal
uuid: "64240eb20c9a4d38862ac49b1f77ec30"
format_stamp: "Formatted at 2026-06-20 23:36:05 on dist-test-slave-tcgx"
server_key: "f01109ecb5c5f41d077c29a5b2141458"
server_key_iv: "f62de12dcbff465eadf9f5a18800be5d"
server_key_version: "encryptionkey@0"
I20260620 23:36:05.389851 4721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:36:05.407938 4721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:36:05.412652 4721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:36:05.412955 4721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:36:05.413976 4721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:36:05.415429 4721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:36:05.415556 4721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:05.415706 4721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:36:05.415795 4721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:05.448951 4721 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37521
I20260620 23:36:05.449013 4847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37521 every 8 connection(s)
I20260620 23:36:05.449877 4721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
I20260620 23:36:05.460049 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 4721
I20260620 23:36:05.460242 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/minicluster-data/ts-1/wal/instance
I20260620 23:36:05.460764 388 external_mini_cluster.cc:1428] Setting key da3b23c69fefde372d56038f983e3e72
Jun 20 23:36:05 dist-test-slave-tcgx krb5kdc[4483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998565, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:36:05.471570 4850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:05.452477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33309 (local address 127.0.97.62:34805)
0620 23:36:05.452930 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:05.452937 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:05.454068 (+ 1131us) server_negotiation.cc:407] Connection header received
0620 23:36:05.456078 (+ 2010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:05.456082 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:05.456153 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:05.456253 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:05.457772 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:05.458492 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:05.460500 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:05.460976 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:05.461077 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:05.465667 (+ 4590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:05.465674 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:05.465677 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:05.465709 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:05.468382 (+ 2673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:05.469212 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:05.469216 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:05.469219 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:05.469346 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:05.469796 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:05.469806 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:05.469808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:05.470068 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:05.470221 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:05.470924 (+ 703us) server_negotiation.cc:299] Negotiation successful
0620 23:36:05.471178 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":208,"threads_started":1}
I20260620 23:36:05.472595 4851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:05.453081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34805 (local address 127.0.97.2:33309)
0620 23:36:05.453886 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:05.453933 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:05.455806 (+ 1873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:05.456503 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:05.456512 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:05.457053 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:05.457615 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:05.457631 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:05.458624 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:05.458627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:05.460344 (+ 1717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:05.460363 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:05.461217 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:05.461224 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:05.461487 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:05.462296 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:05.462321 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:05.465425 (+ 3104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:36:05.468544 (+ 3119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:05.468557 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:05.468582 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:05.469056 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:05.469469 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:05.469475 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:05.469479 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:05.469668 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:05.470205 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:36:05.470214 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:36:05.470567 (+ 353us) client_negotiation.cc:769] Sending connection context
0620 23:36:05.470837 (+ 270us) client_negotiation.cc:240] Negotiation successful
0620 23:36:05.471435 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":309,"threads_started":1}
I20260620 23:36:05.474203 4848 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34805
I20260620 23:36:05.474527 4848 heartbeater.cc:464] Registering TS with master...
I20260620 23:36:05.475251 4848 heartbeater.cc:511] Master 127.0.97.62:34805 requested a full tablet report, sending...
I20260620 23:36:05.476907 4527 ts_manager.cc:194] Registered new tserver with Master: 64240eb20c9a4d38862ac49b1f77ec30 (127.0.97.2:37521)
I20260620 23:36:05.478231 4527 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:33309
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 23:36:05.506891 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34805
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:36:05.839619 4854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:36:05.840060 4854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:36:05.840250 4854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:36:05.847882 4854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:36:05.848001 4854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:36:05.848230 4854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 23:36:05.858345 4854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34805
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:36:05.859740 4854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:36:05.861070 4854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:36:05.874466 4859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:05.874440 4862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:05.874516 4860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:36:05.875227 4854 server_base.cc:1034] running on GCE node
I20260620 23:36:05.875922 4854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:36:05.876952 4854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:36:05.878186 4854 hybrid_clock.cc:648] HybridClock initialized: now 1781998565878136 us; error 47 us; skew 500 ppm
I20260620 23:36:05.878531 4854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:36:05 dist-test-slave-tcgx krb5kdc[4483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998565, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:36:05.881668 4854 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 23:36:05.882874 4854 fs_manager.cc:359] Metadata directory not provided
I20260620 23:36:05.883033 4854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:36:05.883369 4854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:36:05.886338 4854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/minicluster-data/ts-2/data/instance:
uuid: "af7891d7fefc402b9468107474a27d28"
format_stamp: "Formatted at 2026-06-20 23:36:05 on dist-test-slave-tcgx"
server_key: "271b704a75cb65d203ea9060df04c806"
server_key_iv: "dc1834c04cdccee7e88288094974bc06"
server_key_version: "encryptionkey@0"
I20260620 23:36:05.887256 4854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/minicluster-data/ts-2/wal/instance:
uuid: "af7891d7fefc402b9468107474a27d28"
format_stamp: "Formatted at 2026-06-20 23:36:05 on dist-test-slave-tcgx"
server_key: "271b704a75cb65d203ea9060df04c806"
server_key_iv: "dc1834c04cdccee7e88288094974bc06"
server_key_version: "encryptionkey@0"
I20260620 23:36:05.892540 4854 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 23:36:05.896284 4868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:05.897651 4854 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:36:05.897841 4854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/minicluster-data/ts-2/wal
uuid: "af7891d7fefc402b9468107474a27d28"
format_stamp: "Formatted at 2026-06-20 23:36:05 on dist-test-slave-tcgx"
server_key: "271b704a75cb65d203ea9060df04c806"
server_key_iv: "dc1834c04cdccee7e88288094974bc06"
server_key_version: "encryptionkey@0"
I20260620 23:36:05.898080 4854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:36:05.915529 4854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:36:05.920113 4854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:36:05.920403 4854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:36:05.921379 4854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:36:05.922704 4854 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:36:05.922821 4854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:05.922979 4854 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:36:05.923071 4854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260620 23:36:05.956846 4854 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35359
I20260620 23:36:05.956889 4980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35359 every 8 connection(s)
I20260620 23:36:05.957719 4854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
I20260620 23:36:05.959239 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 4854
I20260620 23:36:05.959435 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/minicluster-data/ts-2/wal/instance
I20260620 23:36:05.959867 388 external_mini_cluster.cc:1428] Setting key 0d315a605fe14ff829c0ba4af52ee22c
I20260620 23:36:05.962766 4715 heartbeater.cc:503] Master 127.0.97.62:34805 was elected leader, sending a full tablet report...
Jun 20 23:36:05 dist-test-slave-tcgx krb5kdc[4483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998565, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:36:05.979290 4850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:05.960517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48613 (local address 127.0.97.62:34805)
0620 23:36:05.960730 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:05.960737 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:05.962325 (+ 1588us) server_negotiation.cc:407] Connection header received
0620 23:36:05.964798 (+ 2473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:05.964804 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:05.964901 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:05.965014 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:05.967067 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:05.967808 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:05.969401 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:05.969842 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:05.969978 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:05.973452 (+ 3474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:05.973459 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:05.973464 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:05.973506 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:05.976208 (+ 2702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:05.976971 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:05.976981 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:05.976984 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:05.977097 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:05.977532 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:05.977537 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:05.977541 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:05.977823 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:05.977974 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:05.978695 (+ 721us) server_negotiation.cc:299] Negotiation successful
0620 23:36:05.978928 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 23:36:05.980270 4983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:05.961059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34805 (local address 127.0.97.3:48613)
0620 23:36:05.962175 (+ 1116us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:05.962222 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:05.964532 (+ 2310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:05.965246 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:05.965254 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:05.965878 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:05.966840 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:05.966868 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:05.967959 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:05.967965 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:05.969239 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:05.969250 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:05.970071 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:05.970078 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:05.970323 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:05.971030 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:05.971054 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:05.973220 (+ 2166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:36:05.976368 (+ 3148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:05.976377 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:05.976396 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:05.976842 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:05.977208 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:05.977213 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:05.977217 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:05.977414 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:05.977948 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:36:05.977956 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:36:05.978324 (+ 368us) client_negotiation.cc:769] Sending connection context
0620 23:36:05.978596 (+ 272us) client_negotiation.cc:240] Negotiation successful
0620 23:36:05.979181 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":817,"thread_start_us":560,"threads_started":1}
I20260620 23:36:05.981786 4981 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34805
I20260620 23:36:05.982077 4981 heartbeater.cc:464] Registering TS with master...
I20260620 23:36:05.982712 4981 heartbeater.cc:511] Master 127.0.97.62:34805 requested a full tablet report, sending...
I20260620 23:36:05.984259 4527 ts_manager.cc:194] Registered new tserver with Master: af7891d7fefc402b9468107474a27d28 (127.0.97.3:35359)
I20260620 23:36:05.985107 4527 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:48613
I20260620 23:36:05.993846 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:36:06.015544 4850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:05.998727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36522 (local address 127.0.97.62:34805)
0620 23:36:05.998895 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:05.998899 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:05.999691 (+ 792us) server_negotiation.cc:407] Connection header received
0620 23:36:05.999852 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:05.999861 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:05.999963 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:06.000091 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:06.001638 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:06.003173 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:06.004572 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:06.005401 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:06.005567 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:06.008013 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:06.008022 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:06.008037 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:06.008103 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:06.012289 (+ 4186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:06.013370 (+ 1081us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:06.013379 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:06.013386 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:06.013611 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:06.014088 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:06.014092 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:06.014094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:06.014407 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:06.014622 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:06.015094 (+ 472us) server_negotiation.cc:299] Negotiation successful
0620 23:36:06.015262 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 23:36:06.020218 4527 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36522:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:36:06.027056 4527 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:36:06.066457 4993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:06.040575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35359 (local address 127.0.0.1:38152)
0620 23:36:06.054991 (+ 14416us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:06.055016 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:06.055169 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:06.056335 (+ 1166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:06.056342 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:06.056369 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:36:06.056771 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:06.056785 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:06.059423 (+ 2638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:06.059432 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:06.062592 (+ 3160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:06.062610 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:06.065389 (+ 2779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:06.065398 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:06.065592 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:06.065644 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 23:36:06.065772 (+ 128us) client_negotiation.cc:240] Negotiation successful
0620 23:36:06.065992 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14275,"mutex_wait_us":1,"thread_start_us":10313,"threads_started":1}
I20260620 23:36:06.067044 4995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:06.039795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37521 (local address 127.0.0.1:57630)
0620 23:36:06.051426 (+ 11631us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:06.051491 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:06.051781 (+ 290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:06.057791 (+ 6010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:06.057799 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:06.057832 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:36:06.058247 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:06.058261 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:06.060505 (+ 2244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:06.060513 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:06.063258 (+ 2745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:06.063278 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:06.065864 (+ 2586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:06.065874 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:06.066124 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:06.066176 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 23:36:06.066248 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 23:36:06.066431 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11258,"mutex_wait_us":35,"thread_start_us":15473,"threads_started":1}
I20260620 23:36:06.068017 4996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:06.040967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38152 (local address 127.0.97.3:35359)
0620 23:36:06.050446 (+ 9479us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:06.050461 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:06.055133 (+ 4672us) server_negotiation.cc:407] Connection header received
0620 23:36:06.055335 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:06.055344 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:06.055650 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:06.055973 (+ 323us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:36:06.057020 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:06.059225 (+ 2205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:06.062830 (+ 3605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:06.064809 (+ 1979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:06.064961 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:06.065259 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:06.067327 (+ 2068us) server_negotiation.cc:299] Negotiation successful
0620 23:36:06.067550 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9224,"thread_start_us":9496,"threads_started":1}
I20260620 23:36:06.068195 4994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:06.040112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57630 (local address 127.0.97.2:37521)
0620 23:36:06.056701 (+ 16589us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:06.056712 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:06.056746 (+ 34us) server_negotiation.cc:407] Connection header received
0620 23:36:06.056848 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:06.056857 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:06.057131 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:06.057333 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:36:06.058477 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:06.060332 (+ 1855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:06.063503 (+ 3171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:06.065606 (+ 2103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:06.067284 (+ 1678us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:06.067539 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:06.067633 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 23:36:06.067827 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16385,"thread_start_us":17402,"threads_started":1}
I20260620 23:36:06.072405 4916 tablet_service.cc:1467] Processing CreateTablet for tablet 02ae54a76a9e4b569715fc4a3b98025d (DEFAULT_TABLE table=test-table [id=78d5dbd68be14b4bab158370a9d895a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:36:06.072959 4783 tablet_service.cc:1467] Processing CreateTablet for tablet 02ae54a76a9e4b569715fc4a3b98025d (DEFAULT_TABLE table=test-table [id=78d5dbd68be14b4bab158370a9d895a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:36:06.074076 4916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02ae54a76a9e4b569715fc4a3b98025d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:36:06.074481 4783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02ae54a76a9e4b569715fc4a3b98025d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:36:06.081187 4993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:06.062486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42509 (local address 127.0.0.1:46630)
0620 23:36:06.067041 (+ 4555us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:06.067064 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:06.067170 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:06.072177 (+ 5007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:06.072182 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:06.072208 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:36:06.072390 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:06.072403 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:06.074681 (+ 2278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:06.074687 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:06.076593 (+ 1906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:06.076610 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:06.080358 (+ 3748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:06.080365 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:06.080568 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:06.080603 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 23:36:06.080699 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 23:36:06.080859 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4400,"thread_start_us":6494,"threads_started":1}
I20260620 23:36:06.082437 4997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:06.055960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46630 (local address 127.0.97.1:42509)
0620 23:36:06.069769 (+ 13809us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:06.069778 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:06.069810 (+ 32us) server_negotiation.cc:407] Connection header received
0620 23:36:06.071428 (+ 1618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:06.071437 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:06.071688 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:06.071891 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:36:06.072572 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:06.074458 (+ 1886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:06.078151 (+ 3693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:06.079847 (+ 1696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:06.079998 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:06.080239 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:06.081810 (+ 1571us) server_negotiation.cc:299] Negotiation successful
0620 23:36:06.081996 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13585,"thread_start_us":15120,"threads_started":1}
I20260620 23:36:06.086956 4650 tablet_service.cc:1467] Processing CreateTablet for tablet 02ae54a76a9e4b569715fc4a3b98025d (DEFAULT_TABLE table=test-table [id=78d5dbd68be14b4bab158370a9d895a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:36:06.088498 4650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02ae54a76a9e4b569715fc4a3b98025d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:36:06.095098 5000 tablet_bootstrap.cc:492] T 02ae54a76a9e4b569715fc4a3b98025d P 64240eb20c9a4d38862ac49b1f77ec30: Bootstrap starting.
I20260620 23:36:06.099768 4999 tablet_bootstrap.cc:492] T 02ae54a76a9e4b569715fc4a3b98025d P af7891d7fefc402b9468107474a27d28: Bootstrap starting.
I20260620 23:36:06.104691 5000 tablet_bootstrap.cc:654] T 02ae54a76a9e4b569715fc4a3b98025d P 64240eb20c9a4d38862ac49b1f77ec30: Neither blocks nor log segments found. Creating new log.
I20260620 23:36:06.104957 4999 tablet_bootstrap.cc:654] T 02ae54a76a9e4b569715fc4a3b98025d P af7891d7fefc402b9468107474a27d28: Neither blocks nor log segments found. Creating new log.
I20260620 23:36:06.106259 4999 log.cc:824] T 02ae54a76a9e4b569715fc4a3b98025d P af7891d7fefc402b9468107474a27d28: Log is configured to *not* fsync() on all Append() calls
I20260620 23:36:06.106297 5000 log.cc:824] T 02ae54a76a9e4b569715fc4a3b98025d P 64240eb20c9a4d38862ac49b1f77ec30: Log is configured to *not* fsync() on all Append() calls
I20260620 23:36:06.113759 5001 tablet_bootstrap.cc:492] T 02ae54a76a9e4b569715fc4a3b98025d P 56d963cb3a8544a88c29e0f233172e98: Bootstrap starting.
I20260620 23:36:06.116370 5000 tablet_bootstrap.cc:492] T 02ae54a76a9e4b569715fc4a3b98025d P 64240eb20c9a4d38862ac49b1f77ec30: No bootstrap required, opened a new log
I20260620 23:36:06.116780 5000 ts_tablet_manager.cc:1397] T 02ae54a76a9e4b569715fc4a3b98025d P 64240eb20c9a4d38862ac49b1f77ec30: Time spent bootstrapping tablet: real 0.022s user 0.012s sys 0.003s
I20260620 23:36:06.120781 4999 tablet_bootstrap.cc:492] T 02ae54a76a9e4b569715fc4a3b98025d P af7891d7fefc402b9468107474a27d28: No bootstrap required, opened a new log
I20260620 23:36:06.121124 4999 ts_tablet_manager.cc:1397] T 02ae54a76a9e4b569715fc4a3b98025d P af7891d7fefc402b9468107474a27d28: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.009s
I20260620 23:36:06.121284 5001 tablet_bootstrap.cc:654] T 02ae54a76a9e4b569715fc4a3b98025d P 56d963cb3a8544a88c29e0f233172e98: Neither blocks nor log segments found. Creating new log.
I20260620 23:36:06.122785 5001 log.cc:824] T 02ae54a76a9e4b569715fc4a3b98025d P 56d963cb3a8544a88c29e0f233172e98: Log is configured to *not* fsync() on all Append() calls
I20260620 23:36:06.130683 5000 raft_consensus.cc:348] T 02ae54a76a9e4b569715fc4a3b98025d P 64240eb20c9a4d38862ac49b1f77ec30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64240eb20c9a4d38862ac49b1f77ec30" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37521 } } peers { permanent_uuid: "af7891d7fefc402b9468107474a27d28" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35359 } } peers { permanent_uuid: "56d963cb3a8544a88c29e0f233172e98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42509 } }
I20260620 23:36:06.131322 5000 raft_consensus.cc:374] T 02ae54a76a9e4b569715fc4a3b98025d P 64240eb20c9a4d38862ac49b1f77ec30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:36:06.131469 5000 raft_consensus.cc:729] T 02ae54a76a9e4b569715fc4a3b98025d P 64240eb20c9a4d38862ac49b1f77ec30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64240eb20c9a4d38862ac49b1f77ec30, State: Initialized, Role: FOLLOWER
I20260620 23:36:06.132360 5000 consensus_queue.cc:260] T 02ae54a76a9e4b569715fc4a3b98025d P 64240eb20c9a4d38862ac49b1f77ec30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64240eb20c9a4d38862ac49b1f77ec30" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37521 } } peers { permanent_uuid: "af7891d7fefc402b9468107474a27d28" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35359 } } peers { permanent_uuid: "56d963cb3a8544a88c29e0f233172e98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42509 } }
I20260620 23:36:06.133955 4848 heartbeater.cc:503] Master 127.0.97.62:34805 was elected leader, sending a full tablet report...
I20260620 23:36:06.134349 4999 raft_consensus.cc:348] T 02ae54a76a9e4b569715fc4a3b98025d P af7891d7fefc402b9468107474a27d28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64240eb20c9a4d38862ac49b1f77ec30" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37521 } } peers { permanent_uuid: "af7891d7fefc402b9468107474a27d28" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35359 } } peers { permanent_uuid: "56d963cb3a8544a88c29e0f233172e98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42509 } }
I20260620 23:36:06.134773 4999 raft_consensus.cc:374] T 02ae54a76a9e4b569715fc4a3b98025d P af7891d7fefc402b9468107474a27d28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:36:06.134884 4999 raft_consensus.cc:729] T 02ae54a76a9e4b569715fc4a3b98025d P af7891d7fefc402b9468107474a27d28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af7891d7fefc402b9468107474a27d28, State: Initialized, Role: FOLLOWER
I20260620 23:36:06.135272 5000 ts_tablet_manager.cc:1428] T 02ae54a76a9e4b569715fc4a3b98025d P 64240eb20c9a4d38862ac49b1f77ec30: Time spent starting tablet: real 0.018s user 0.018s sys 0.001s
I20260620 23:36:06.135536 4999 consensus_queue.cc:260] T 02ae54a76a9e4b569715fc4a3b98025d P af7891d7fefc402b9468107474a27d28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64240eb20c9a4d38862ac49b1f77ec30" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37521 } } peers { permanent_uuid: "af7891d7fefc402b9468107474a27d28" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35359 } } peers { permanent_uuid: "56d963cb3a8544a88c29e0f233172e98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42509 } }
I20260620 23:36:06.136662 5001 tablet_bootstrap.cc:492] T 02ae54a76a9e4b569715fc4a3b98025d P 56d963cb3a8544a88c29e0f233172e98: No bootstrap required, opened a new log
I20260620 23:36:06.137100 5001 ts_tablet_manager.cc:1397] T 02ae54a76a9e4b569715fc4a3b98025d P 56d963cb3a8544a88c29e0f233172e98: Time spent bootstrapping tablet: real 0.024s user 0.014s sys 0.004s
I20260620 23:36:06.144520 4981 heartbeater.cc:503] Master 127.0.97.62:34805 was elected leader, sending a full tablet report...
I20260620 23:36:06.145268 4999 ts_tablet_manager.cc:1428] T 02ae54a76a9e4b569715fc4a3b98025d P af7891d7fefc402b9468107474a27d28: Time spent starting tablet: real 0.024s user 0.014s sys 0.012s
I20260620 23:36:06.154268 5001 raft_consensus.cc:348] T 02ae54a76a9e4b569715fc4a3b98025d P 56d963cb3a8544a88c29e0f233172e98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64240eb20c9a4d38862ac49b1f77ec30" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37521 } } peers { permanent_uuid: "af7891d7fefc402b9468107474a27d28" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35359 } } peers { permanent_uuid: "56d963cb3a8544a88c29e0f233172e98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42509 } }
I20260620 23:36:06.154886 5001 raft_consensus.cc:374] T 02ae54a76a9e4b569715fc4a3b98025d P 56d963cb3a8544a88c29e0f233172e98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:36:06.155037 5001 raft_consensus.cc:729] T 02ae54a76a9e4b569715fc4a3b98025d P 56d963cb3a8544a88c29e0f233172e98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56d963cb3a8544a88c29e0f233172e98, State: Initialized, Role: FOLLOWER
I20260620 23:36:06.155825 5001 consensus_queue.cc:260] T 02ae54a76a9e4b569715fc4a3b98025d P 56d963cb3a8544a88c29e0f233172e98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64240eb20c9a4d38862ac49b1f77ec30" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37521 } } peers { permanent_uuid: "af7891d7fefc402b9468107474a27d28" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35359 } } peers { permanent_uuid: "56d963cb3a8544a88c29e0f233172e98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42509 } }
I20260620 23:36:06.157891 5001 ts_tablet_manager.cc:1428] T 02ae54a76a9e4b569715fc4a3b98025d P 56d963cb3a8544a88c29e0f233172e98: Time spent starting tablet: real 0.021s user 0.010s sys 0.007s
W20260620 23:36:06.178767 4716 tablet.cc:2369] T 02ae54a76a9e4b569715fc4a3b98025d P 56d963cb3a8544a88c29e0f233172e98: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:36:06.201851 4849 tablet.cc:2369] T 02ae54a76a9e4b569715fc4a3b98025d P 64240eb20c9a4d38862ac49b1f77ec30: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:36:06.209203 4982 tablet.cc:2369] T 02ae54a76a9e4b569715fc4a3b98025d P af7891d7fefc402b9468107474a27d28: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:36:06.226312 5007 raft_consensus.cc:482] T 02ae54a76a9e4b569715fc4a3b98025d P 56d963cb3a8544a88c29e0f233172e98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:36:06.226641 5007 raft_consensus.cc:504] T 02ae54a76a9e4b569715fc4a3b98025d P 56d963cb3a8544a88c29e0f233172e98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64240eb20c9a4d38862ac49b1f77ec30" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37521 } } peers { permanent_uuid: "af7891d7fefc402b9468107474a27d28" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35359 } } peers { permanent_uuid: "56d963cb3a8544a88c29e0f233172e98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42509 } }
I20260620 23:36:06.228510 5007 leader_election.cc:288] T 02ae54a76a9e4b569715fc4a3b98025d P 56d963cb3a8544a88c29e0f233172e98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64240eb20c9a4d38862ac49b1f77ec30 (127.0.97.2:37521), af7891d7fefc402b9468107474a27d28 (127.0.97.3:35359)
I20260620 23:36:06.237426 4994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:06.229042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44147 (local address 127.0.97.2:37521)
0620 23:36:06.229486 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:06.229495 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:06.230820 (+ 1325us) server_negotiation.cc:407] Connection header received
0620 23:36:06.231043 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:06.231048 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:06.231118 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:06.231221 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:36:06.232129 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:06.233168 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:06.235512 (+ 2344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:06.236336 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:06.236471 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:06.236572 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:06.237015 (+ 443us) server_negotiation.cc:299] Negotiation successful
0620 23:36:06.237150 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272}
I20260620 23:36:06.237656 5008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:06.229033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37521 (local address 127.0.97.1:44147)
0620 23:36:06.230693 (+ 1660us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:06.230719 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:06.230868 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:06.231249 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:06.231256 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:06.231288 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:36:06.231934 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:06.231951 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:06.233345 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:06.233350 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:06.235350 (+ 2000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:06.235362 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:06.236572 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:06.236581 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:06.236803 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:06.236857 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 23:36:06.236977 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 23:36:06.237152 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1497,"mutex_wait_us":125,"thread_start_us":888,"threads_started":1}
I20260620 23:36:06.238525 4996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:06.229391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40435 (local address 127.0.97.3:35359)
0620 23:36:06.229643 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:06.229652 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:06.230468 (+ 816us) server_negotiation.cc:407] Connection header received
0620 23:36:06.230756 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:06.230762 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:06.230874 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:06.231023 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:36:06.231996 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:06.233134 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:06.236329 (+ 3195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:06.237372 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:06.237499 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:06.237604 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:06.238156 (+ 552us) server_negotiation.cc:299] Negotiation successful
0620 23:36:06.238281 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 23:36:06.238672 5009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:06.228859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35359 (local address 127.0.97.1:40435)
0620 23:36:06.230329 (+ 1470us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:06.230355 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:06.230598 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:06.231340 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:06.231343 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:06.231367 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:36:06.231844 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:06.231865 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:06.233280 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:06.233287 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:06.236148 (+ 2861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:06.236166 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:06.237780 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:06.237786 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:06.237980 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:06.238010 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 23:36:06.238131 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 23:36:06.238271 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1296,"mutex_wait_us":56,"thread_start_us":1489,"threads_started":1}
I20260620 23:36:06.238972 4803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02ae54a76a9e4b569715fc4a3b98025d" candidate_uuid: "56d963cb3a8544a88c29e0f233172e98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64240eb20c9a4d38862ac49b1f77ec30" is_pre_election: true
I20260620 23:36:06.239602 4803 raft_consensus.cc:2454] T 02ae54a76a9e4b569715fc4a3b98025d P 64240eb20c9a4d38862ac49b1f77ec30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56d963cb3a8544a88c29e0f233172e98 in term 0.
I20260620 23:36:06.239611 4936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02ae54a76a9e4b569715fc4a3b98025d" candidate_uuid: "56d963cb3a8544a88c29e0f233172e98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af7891d7fefc402b9468107474a27d28" is_pre_election: true
I20260620 23:36:06.240036 4936 raft_consensus.cc:2454] T 02ae54a76a9e4b569715fc4a3b98025d P af7891d7fefc402b9468107474a27d28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56d963cb3a8544a88c29e0f233172e98 in term 0.
I20260620 23:36:06.240424 4604 leader_election.cc:302] T 02ae54a76a9e4b569715fc4a3b98025d P 56d963cb3a8544a88c29e0f233172e98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56d963cb3a8544a88c29e0f233172e98, 64240eb20c9a4d38862ac49b1f77ec30; no voters:
I20260620 23:36:06.240971 5007 raft_consensus.cc:2781] T 02ae54a76a9e4b569715fc4a3b98025d P 56d963cb3a8544a88c29e0f233172e98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:36:06.241154 5007 raft_consensus.cc:482] T 02ae54a76a9e4b569715fc4a3b98025d P 56d963cb3a8544a88c29e0f233172e98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:36:06.241282 5007 raft_consensus.cc:3037] T 02ae54a76a9e4b569715fc4a3b98025d P 56d963cb3a8544a88c29e0f233172e98 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:36:06.245464 5007 raft_consensus.cc:504] T 02ae54a76a9e4b569715fc4a3b98025d P 56d963cb3a8544a88c29e0f233172e98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64240eb20c9a4d38862ac49b1f77ec30" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37521 } } peers { permanent_uuid: "af7891d7fefc402b9468107474a27d28" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35359 } } peers { permanent_uuid: "56d963cb3a8544a88c29e0f233172e98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42509 } }
I20260620 23:36:06.246376 5007 leader_election.cc:288] T 02ae54a76a9e4b569715fc4a3b98025d P 56d963cb3a8544a88c29e0f233172e98 [CANDIDATE]: Term 1 election: Requested vote from peers 64240eb20c9a4d38862ac49b1f77ec30 (127.0.97.2:37521), af7891d7fefc402b9468107474a27d28 (127.0.97.3:35359)
I20260620 23:36:06.247009 4803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02ae54a76a9e4b569715fc4a3b98025d" candidate_uuid: "56d963cb3a8544a88c29e0f233172e98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64240eb20c9a4d38862ac49b1f77ec30"
I20260620 23:36:06.247284 4803 raft_consensus.cc:3037] T 02ae54a76a9e4b569715fc4a3b98025d P 64240eb20c9a4d38862ac49b1f77ec30 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:36:06.249384 4936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02ae54a76a9e4b569715fc4a3b98025d" candidate_uuid: "56d963cb3a8544a88c29e0f233172e98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af7891d7fefc402b9468107474a27d28"
I20260620 23:36:06.249617 4936 raft_consensus.cc:3037] T 02ae54a76a9e4b569715fc4a3b98025d P af7891d7fefc402b9468107474a27d28 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:36:06.251207 4803 raft_consensus.cc:2454] T 02ae54a76a9e4b569715fc4a3b98025d P 64240eb20c9a4d38862ac49b1f77ec30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56d963cb3a8544a88c29e0f233172e98 in term 1.
I20260620 23:36:06.251782 4604 leader_election.cc:302] T 02ae54a76a9e4b569715fc4a3b98025d P 56d963cb3a8544a88c29e0f233172e98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56d963cb3a8544a88c29e0f233172e98, 64240eb20c9a4d38862ac49b1f77ec30; no voters:
I20260620 23:36:06.252118 5007 raft_consensus.cc:2781] T 02ae54a76a9e4b569715fc4a3b98025d P 56d963cb3a8544a88c29e0f233172e98 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:36:06.252252 4936 raft_consensus.cc:2454] T 02ae54a76a9e4b569715fc4a3b98025d P af7891d7fefc402b9468107474a27d28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56d963cb3a8544a88c29e0f233172e98 in term 1.
I20260620 23:36:06.252718 5007 raft_consensus.cc:686] T 02ae54a76a9e4b569715fc4a3b98025d P 56d963cb3a8544a88c29e0f233172e98 [term 1 LEADER]: Becoming Leader. State: Replica: 56d963cb3a8544a88c29e0f233172e98, State: Running, Role: LEADER
I20260620 23:36:06.253489 5007 consensus_queue.cc:237] T 02ae54a76a9e4b569715fc4a3b98025d P 56d963cb3a8544a88c29e0f233172e98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64240eb20c9a4d38862ac49b1f77ec30" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37521 } } peers { permanent_uuid: "af7891d7fefc402b9468107474a27d28" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35359 } } peers { permanent_uuid: "56d963cb3a8544a88c29e0f233172e98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42509 } }
I20260620 23:36:06.258862 4527 catalog_manager.cc:5495] T 02ae54a76a9e4b569715fc4a3b98025d P 56d963cb3a8544a88c29e0f233172e98 reported cstate change: term changed from 0 to 1, leader changed from <none> to 56d963cb3a8544a88c29e0f233172e98 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "56d963cb3a8544a88c29e0f233172e98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64240eb20c9a4d38862ac49b1f77ec30" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af7891d7fefc402b9468107474a27d28" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56d963cb3a8544a88c29e0f233172e98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42509 } health_report { overall_health: HEALTHY } } }
I20260620 23:36:06.315310 4997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:06.308974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46642 (local address 127.0.97.1:42509)
0620 23:36:06.309200 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:06.309207 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:06.309396 (+ 189us) server_negotiation.cc:407] Connection header received
0620 23:36:06.309579 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:06.309585 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:06.309679 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:06.309838 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:36:06.310180 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:06.311333 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:06.312676 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:06.313285 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:06.313490 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:06.314018 (+ 528us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:36:06.314725 (+ 707us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:36:06.314867 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:06.314967 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 23:36:06.315054 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 23:36:06.323844 4803 raft_consensus.cc:1261] T 02ae54a76a9e4b569715fc4a3b98025d P 64240eb20c9a4d38862ac49b1f77ec30 [term 1 FOLLOWER]: Refusing update from remote peer 56d963cb3a8544a88c29e0f233172e98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:36:06.323859 4936 raft_consensus.cc:1261] T 02ae54a76a9e4b569715fc4a3b98025d P af7891d7fefc402b9468107474a27d28 [term 1 FOLLOWER]: Refusing update from remote peer 56d963cb3a8544a88c29e0f233172e98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:36:06.325055 5010 consensus_queue.cc:1035] T 02ae54a76a9e4b569715fc4a3b98025d P 56d963cb3a8544a88c29e0f233172e98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64240eb20c9a4d38862ac49b1f77ec30" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:36:06.325512 5007 consensus_queue.cc:1035] T 02ae54a76a9e4b569715fc4a3b98025d P 56d963cb3a8544a88c29e0f233172e98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af7891d7fefc402b9468107474a27d28" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:36:06.334831 5013 mvcc.cc:205] Tried to move back new op lower bound from 7299066127651483648 to 7299066127380094976. Current Snapshot: MvccSnapshot[applied={T|T < 7299066127651483648}]
I20260620 23:36:06.335667 5015 mvcc.cc:205] Tried to move back new op lower bound from 7299066127651483648 to 7299066127380094976. Current Snapshot: MvccSnapshot[applied={T|T < 7299066127651483648}]
I20260620 23:36:06.352178 4527 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:36:06.352484 4527 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:36:06.357146 4527 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a5d93bbd3950430a824ef421660b46ac: Sending DeleteTablet for 3 replicas of tablet 02ae54a76a9e4b569715fc4a3b98025d
I20260620 23:36:06.358515 4782 tablet_service.cc:1514] Processing DeleteTablet for tablet 02ae54a76a9e4b569715fc4a3b98025d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:36:06 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57630
I20260620 23:36:06.358734 4916 tablet_service.cc:1514] Processing DeleteTablet for tablet 02ae54a76a9e4b569715fc4a3b98025d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:36:06 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38152
I20260620 23:36:06.358999 4650 tablet_service.cc:1514] Processing DeleteTablet for tablet 02ae54a76a9e4b569715fc4a3b98025d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:36:06 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46630
I20260620 23:36:06.359163 4783 tablet_service.cc:1514] Processing DeleteTablet for tablet 02ae54a76a9e4b569715fc4a3b98025d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:36:06 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57630
I20260620 23:36:06.360719 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 4588
I20260620 23:36:06.369517 5024 tablet_replica.cc:330] T 02ae54a76a9e4b569715fc4a3b98025d P af7891d7fefc402b9468107474a27d28: stopping tablet replica
I20260620 23:36:06.369949 5024 raft_consensus.cc:2229] T 02ae54a76a9e4b569715fc4a3b98025d P af7891d7fefc402b9468107474a27d28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:36:06.370414 5024 raft_consensus.cc:2258] T 02ae54a76a9e4b569715fc4a3b98025d P af7891d7fefc402b9468107474a27d28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:36:06.372641 5023 tablet_replica.cc:330] T 02ae54a76a9e4b569715fc4a3b98025d P 64240eb20c9a4d38862ac49b1f77ec30: stopping tablet replica
I20260620 23:36:06.373024 5023 raft_consensus.cc:2229] T 02ae54a76a9e4b569715fc4a3b98025d P 64240eb20c9a4d38862ac49b1f77ec30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:36:06.373476 5023 raft_consensus.cc:2258] T 02ae54a76a9e4b569715fc4a3b98025d P 64240eb20c9a4d38862ac49b1f77ec30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:36:06.375648 5023 ts_tablet_manager.cc:1905] T 02ae54a76a9e4b569715fc4a3b98025d P 64240eb20c9a4d38862ac49b1f77ec30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:36:06.375648 5024 ts_tablet_manager.cc:1905] T 02ae54a76a9e4b569715fc4a3b98025d P af7891d7fefc402b9468107474a27d28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:36:06.382805 5024 ts_tablet_manager.cc:1918] T 02ae54a76a9e4b569715fc4a3b98025d P af7891d7fefc402b9468107474a27d28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:36:06.382805 5023 ts_tablet_manager.cc:1918] T 02ae54a76a9e4b569715fc4a3b98025d P 64240eb20c9a4d38862ac49b1f77ec30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:36:06.383186 5024 log.cc:1196] T 02ae54a76a9e4b569715fc4a3b98025d P af7891d7fefc402b9468107474a27d28: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/minicluster-data/ts-2/wal/wals/02ae54a76a9e4b569715fc4a3b98025d
I20260620 23:36:06.383589 5023 log.cc:1196] T 02ae54a76a9e4b569715fc4a3b98025d P 64240eb20c9a4d38862ac49b1f77ec30: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/minicluster-data/ts-1/wal/wals/02ae54a76a9e4b569715fc4a3b98025d
I20260620 23:36:06.383739 5024 ts_tablet_manager.cc:1939] T 02ae54a76a9e4b569715fc4a3b98025d P af7891d7fefc402b9468107474a27d28: Deleting consensus metadata
I20260620 23:36:06.384235 5023 ts_tablet_manager.cc:1939] T 02ae54a76a9e4b569715fc4a3b98025d P 64240eb20c9a4d38862ac49b1f77ec30: Deleting consensus metadata
I20260620 23:36:06.385825 4514 catalog_manager.cc:4841] TS af7891d7fefc402b9468107474a27d28 (127.0.97.3:35359): tablet 02ae54a76a9e4b569715fc4a3b98025d (table test-table [id=78d5dbd68be14b4bab158370a9d895a6]) successfully deleted
I20260620 23:36:06.385932 4512 catalog_manager.cc:4841] TS 64240eb20c9a4d38862ac49b1f77ec30 (127.0.97.2:37521): tablet 02ae54a76a9e4b569715fc4a3b98025d (table test-table [id=78d5dbd68be14b4bab158370a9d895a6]) successfully deleted
W20260620 23:36:06.387023 4512 catalog_manager.cc:4808] TS 64240eb20c9a4d38862ac49b1f77ec30 (127.0.97.2:37521): delete failed for tablet 02ae54a76a9e4b569715fc4a3b98025d because the tablet was not found. No further retry: Not found: Tablet not found: 02ae54a76a9e4b569715fc4a3b98025d
I20260620 23:36:06.387405 5025 tablet_replica.cc:330] T 02ae54a76a9e4b569715fc4a3b98025d P 56d963cb3a8544a88c29e0f233172e98: stopping tablet replica
I20260620 23:36:06.387907 5025 raft_consensus.cc:2229] T 02ae54a76a9e4b569715fc4a3b98025d P 56d963cb3a8544a88c29e0f233172e98 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:36:06.388617 5025 raft_consensus.cc:2258] T 02ae54a76a9e4b569715fc4a3b98025d P 56d963cb3a8544a88c29e0f233172e98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:36:06.390992 5025 ts_tablet_manager.cc:1905] T 02ae54a76a9e4b569715fc4a3b98025d P 56d963cb3a8544a88c29e0f233172e98: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 23:36:06 dist-test-slave-tcgx krb5kdc[4483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 23:36:06.399013 5025 ts_tablet_manager.cc:1918] T 02ae54a76a9e4b569715fc4a3b98025d P 56d963cb3a8544a88c29e0f233172e98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:36:06.399200 5025 log.cc:1196] T 02ae54a76a9e4b569715fc4a3b98025d P 56d963cb3a8544a88c29e0f233172e98: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998525501163-388-0/minicluster-data/ts-0/wal/wals/02ae54a76a9e4b569715fc4a3b98025d
I20260620 23:36:06.399852 5025 ts_tablet_manager.cc:1939] T 02ae54a76a9e4b569715fc4a3b98025d P 56d963cb3a8544a88c29e0f233172e98: Deleting consensus metadata
I20260620 23:36:06.401847 4512 catalog_manager.cc:4841] TS 56d963cb3a8544a88c29e0f233172e98 (127.0.97.1:42509): tablet 02ae54a76a9e4b569715fc4a3b98025d (table test-table [id=78d5dbd68be14b4bab158370a9d895a6]) successfully deleted
I20260620 23:36:06.402169 4997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:06.363372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46652 (local address 127.0.97.1:42509)
0620 23:36:06.363527 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:06.363533 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:06.385509 (+ 21976us) server_negotiation.cc:407] Connection header received
0620 23:36:06.386198 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:06.386204 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:06.386314 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:06.386433 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:06.388174 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:06.389266 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:06.390652 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:06.391284 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:06.391422 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:06.395773 (+ 4351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:06.395786 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:06.395800 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:06.395843 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:06.399164 (+ 3321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:06.399889 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:06.399897 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:06.399901 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:06.400073 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:06.400473 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:06.400478 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:06.400482 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:06.400838 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:06.401181 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:06.401536 (+ 355us) server_negotiation.cc:299] Negotiation successful
0620 23:36:06.401768 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 23:36:06.402801 4710 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:36:06.502547 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 4588
I20260620 23:36:06.522710 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 4721
Jun 20 23:36:06 dist-test-slave-tcgx krb5kdc[4483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 23:36:06.538527 4994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:06.523711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57642 (local address 127.0.97.2:37521)
0620 23:36:06.523918 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:06.523923 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:06.523941 (+ 18us) server_negotiation.cc:407] Connection header received
0620 23:36:06.524041 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:06.524045 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:06.524119 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:06.524204 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:06.525548 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:06.526270 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:06.527777 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:06.528155 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:06.528252 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:06.532836 (+ 4584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:06.532850 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:06.532867 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:06.532913 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:06.535735 (+ 2822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:06.536601 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:06.536609 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:06.536611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:06.536732 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:06.537062 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:06.537065 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:06.537067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:06.537456 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:06.537666 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:06.537953 (+ 287us) server_negotiation.cc:299] Negotiation successful
0620 23:36:06.538104 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260620 23:36:06.539567 4843 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:36:06.630504 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 4721
I20260620 23:36:06.648530 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 4854
Jun 20 23:36:06 dist-test-slave-tcgx krb5kdc[4483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 23:36:06.663838 4996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:06.649324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38168 (local address 127.0.97.3:35359)
0620 23:36:06.649495 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:06.649500 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:06.649518 (+ 18us) server_negotiation.cc:407] Connection header received
0620 23:36:06.649668 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:06.649672 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:06.649738 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:06.649835 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:06.651237 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:06.652074 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:06.653303 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:06.653732 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:06.653843 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:06.658031 (+ 4188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:06.658049 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:06.658066 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:06.658115 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:06.661289 (+ 3174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:06.661965 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:06.661973 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:06.661976 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:06.662122 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:06.662458 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:06.662461 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:06.662463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:06.662824 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:06.663118 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:06.663352 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 23:36:06.663590 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 23:36:06.664525 4976 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:36:06.753180 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 4854
I20260620 23:36:06.771303 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 4496
I20260620 23:36:06.772457 4557 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:36:06.880383 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 4496
2026-06-20T23:36:06Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4328 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T23:36:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:36:06Z Disabled control of system clock
I20260620 23:36:06.951233 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998525501163-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998525501163-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998525501163-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36885
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:37559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 23:36:07.280556 5034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:36:07.280851 5034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:36:07.280937 5034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:36:07.288259 5034 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:36:07.288379 5034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:36:07.288422 5034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:36:07.288475 5034 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:36:07.288518 5034 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:36:07.297888 5034 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998525501163-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998525501163-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36885
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998525501163-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:36:07.299181 5034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:36:07.300952 5034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:36:07.312816 5040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:07.312855 5039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:07.312893 5042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:36:07.313984 5034 server_base.cc:1034] running on GCE node
I20260620 23:36:07.314810 5034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:36:07.316351 5034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:36:07.317557 5034 hybrid_clock.cc:648] HybridClock initialized: now 1781998567317514 us; error 35 us; skew 500 ppm
I20260620 23:36:07.317932 5034 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:36:07.320217 5034 webserver.cc:460] Webserver started at http://127.0.97.62:45299/ using document root <none> and password file <none>
I20260620 23:36:07.320924 5034 fs_manager.cc:359] Metadata directory not provided
I20260620 23:36:07.321034 5034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:36:07.321378 5034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:36:07.324354 5034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998525501163-388-0/minicluster-data/master-0/data/instance:
uuid: "ac882235da6b4bb29c5f469d9415fe09"
format_stamp: "Formatted at 2026-06-20 23:36:07 on dist-test-slave-tcgx"
server_key: "0d6ab73f464db1930e618938396e6623"
server_key_iv: "4e76748e8d463443fdd69beaacd03eeb"
server_key_version: "encryptionkey@0"
I20260620 23:36:07.325162 5034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998525501163-388-0/minicluster-data/master-0/wal/instance:
uuid: "ac882235da6b4bb29c5f469d9415fe09"
format_stamp: "Formatted at 2026-06-20 23:36:07 on dist-test-slave-tcgx"
server_key: "0d6ab73f464db1930e618938396e6623"
server_key_iv: "4e76748e8d463443fdd69beaacd03eeb"
server_key_version: "encryptionkey@0"
I20260620 23:36:07.330305 5034 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 23:36:07.333860 5048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:07.335273 5034 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:36:07.335448 5034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998525501163-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998525501163-388-0/minicluster-data/master-0/wal
uuid: "ac882235da6b4bb29c5f469d9415fe09"
format_stamp: "Formatted at 2026-06-20 23:36:07 on dist-test-slave-tcgx"
server_key: "0d6ab73f464db1930e618938396e6623"
server_key_iv: "4e76748e8d463443fdd69beaacd03eeb"
server_key_version: "encryptionkey@0"
I20260620 23:36:07.335666 5034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998525501163-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998525501163-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998525501163-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:36:07.341581 5034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:36:07.342392 5034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:36:07.342693 5034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:36:07.360787 5034 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36885
I20260620 23:36:07.360809 5099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36885 every 8 connection(s)
I20260620 23:36:07.362217 5034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:36:07.365926 5100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:36:07.371712 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 5034
I20260620 23:36:07.371919 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998525501163-388-0/minicluster-data/master-0/wal/instance
I20260620 23:36:07.372519 388 external_mini_cluster.cc:1428] Setting key 27409d156c679bb9244ba31213444c09
I20260620 23:36:07.377290 5100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac882235da6b4bb29c5f469d9415fe09: Bootstrap starting.
I20260620 23:36:07.381448 5100 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac882235da6b4bb29c5f469d9415fe09: Neither blocks nor log segments found. Creating new log.
I20260620 23:36:07.382520 5103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:07.374722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50288 (local address 127.0.97.62:36885)
0620 23:36:07.375845 (+ 1123us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:07.375862 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:07.375912 (+ 50us) server_negotiation.cc:407] Connection header received
0620 23:36:07.378223 (+ 2311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:07.378252 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:07.378806 (+ 554us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:07.379354 (+ 548us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:07.380406 (+ 1052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:07.380414 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:07.380430 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:36:07.380465 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:07.380544 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:36:07.380566 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:07.380743 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:07.380974 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 23:36:07.381246 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":353,"threads_started":1}
I20260620 23:36:07.382776 5100 log.cc:824] T 00000000000000000000000000000000 P ac882235da6b4bb29c5f469d9415fe09: Log is configured to *not* fsync() on all Append() calls
I20260620 23:36:07.385681 5100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac882235da6b4bb29c5f469d9415fe09: No bootstrap required, opened a new log
I20260620 23:36:07.392871 5100 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac882235da6b4bb29c5f469d9415fe09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac882235da6b4bb29c5f469d9415fe09" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36885 } }
I20260620 23:36:07.393261 5100 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac882235da6b4bb29c5f469d9415fe09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:36:07.393334 5100 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac882235da6b4bb29c5f469d9415fe09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac882235da6b4bb29c5f469d9415fe09, State: Initialized, Role: FOLLOWER
I20260620 23:36:07.393908 5100 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac882235da6b4bb29c5f469d9415fe09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac882235da6b4bb29c5f469d9415fe09" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36885 } }
I20260620 23:36:07.394132 5100 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac882235da6b4bb29c5f469d9415fe09 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:36:07.394228 5100 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac882235da6b4bb29c5f469d9415fe09 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:36:07.394351 5100 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac882235da6b4bb29c5f469d9415fe09 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:36:07.396589 5100 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac882235da6b4bb29c5f469d9415fe09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac882235da6b4bb29c5f469d9415fe09" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36885 } }
I20260620 23:36:07.397074 5100 leader_election.cc:302] T 00000000000000000000000000000000 P ac882235da6b4bb29c5f469d9415fe09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac882235da6b4bb29c5f469d9415fe09; no voters:
I20260620 23:36:07.397689 5100 leader_election.cc:288] T 00000000000000000000000000000000 P ac882235da6b4bb29c5f469d9415fe09 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:36:07.397871 5105 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac882235da6b4bb29c5f469d9415fe09 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:36:07.399096 5105 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac882235da6b4bb29c5f469d9415fe09 [term 1 LEADER]: Becoming Leader. State: Replica: ac882235da6b4bb29c5f469d9415fe09, State: Running, Role: LEADER
I20260620 23:36:07.399755 5105 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac882235da6b4bb29c5f469d9415fe09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac882235da6b4bb29c5f469d9415fe09" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36885 } }
I20260620 23:36:07.400097 5100 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac882235da6b4bb29c5f469d9415fe09 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:36:07.402598 5107 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac882235da6b4bb29c5f469d9415fe09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac882235da6b4bb29c5f469d9415fe09. Latest consensus state: current_term: 1 leader_uuid: "ac882235da6b4bb29c5f469d9415fe09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac882235da6b4bb29c5f469d9415fe09" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36885 } } }
I20260620 23:36:07.403093 5107 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac882235da6b4bb29c5f469d9415fe09 [sys.catalog]: This master's current role is: LEADER
I20260620 23:36:07.403096 5106 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac882235da6b4bb29c5f469d9415fe09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac882235da6b4bb29c5f469d9415fe09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac882235da6b4bb29c5f469d9415fe09" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36885 } } }
I20260620 23:36:07.403532 5106 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac882235da6b4bb29c5f469d9415fe09 [sys.catalog]: This master's current role is: LEADER
I20260620 23:36:07.405014 5114 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:36:07.410694 5114 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:36:07.420055 5114 catalog_manager.cc:1347] Generated new cluster ID: 4108a30c87d14be6a61dc81d2e0deabf
I20260620 23:36:07.420181 5114 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:36:07.435204 5114 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:36:07.435850 5114 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:36:07.451434 5114 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac882235da6b4bb29c5f469d9415fe09: Generated new TSK 0
I20260620 23:36:07.452481 5114 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:36:07.471143 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998525501163-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998525501163-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998525501163-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36885
--builtin_ntp_servers=127.0.97.20:37559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 23:36:07.836804 5124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:36:07.837102 5124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:36:07.837251 5124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:36:07.845124 5124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:36:07.845240 5124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:36:07.845412 5124 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 23:36:07.855219 5124 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998525501163-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998525501163-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998525501163-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:36:07.856570 5124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:36:07.858403 5124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:36:07.872413 5129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:07.872434 5132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:07.872630 5130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:36:07.873119 5124 server_base.cc:1034] running on GCE node
I20260620 23:36:07.873780 5124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:36:07.874817 5124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:36:07.876046 5124 hybrid_clock.cc:648] HybridClock initialized: now 1781998567876002 us; error 89 us; skew 500 ppm
I20260620 23:36:07.876377 5124 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:36:07.878893 5124 webserver.cc:460] Webserver started at http://127.0.97.1:45513/ using document root <none> and password file <none>
I20260620 23:36:07.879698 5124 fs_manager.cc:359] Metadata directory not provided
I20260620 23:36:07.879842 5124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:36:07.880184 5124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:36:07.883584 5124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998525501163-388-0/minicluster-data/ts-0/data/instance:
uuid: "69a230496aab4d7dbd37c889de949355"
format_stamp: "Formatted at 2026-06-20 23:36:07 on dist-test-slave-tcgx"
server_key: "7175c3c37891e02fc5b3a77657928b15"
server_key_iv: "5f15345eceeaf9f17da8f3a41a95d7ff"
server_key_version: "encryptionkey@0"
I20260620 23:36:07.884443 5124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998525501163-388-0/minicluster-data/ts-0/wal/instance:
uuid: "69a230496aab4d7dbd37c889de949355"
format_stamp: "Formatted at 2026-06-20 23:36:07 on dist-test-slave-tcgx"
server_key: "7175c3c37891e02fc5b3a77657928b15"
server_key_iv: "5f15345eceeaf9f17da8f3a41a95d7ff"
server_key_version: "encryptionkey@0"
I20260620 23:36:07.889971 5124 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 23:36:07.893790 5138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:07.895301 5124 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:36:07.895483 5124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998525501163-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998525501163-388-0/minicluster-data/ts-0/wal
uuid: "69a230496aab4d7dbd37c889de949355"
format_stamp: "Formatted at 2026-06-20 23:36:07 on dist-test-slave-tcgx"
server_key: "7175c3c37891e02fc5b3a77657928b15"
server_key_iv: "5f15345eceeaf9f17da8f3a41a95d7ff"
server_key_version: "encryptionkey@0"
I20260620 23:36:07.895725 5124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998525501163-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998525501163-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998525501163-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:36:07.901489 5124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:36:07.902340 5124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:36:07.902606 5124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:36:07.903699 5124 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:36:07.905180 5124 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:36:07.905300 5124 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:07.905429 5124 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:36:07.905521 5124 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:07.938634 5124 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36023
I20260620 23:36:07.938683 5250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36023 every 8 connection(s)
I20260620 23:36:07.940181 5124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
I20260620 23:36:07.946332 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 5124
I20260620 23:36:07.946550 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998525501163-388-0/minicluster-data/ts-0/wal/instance
I20260620 23:36:07.947377 388 external_mini_cluster.cc:1428] Setting key 5b5fe9e952bbca05ef998d5c7db8a13f
I20260620 23:36:07.950235 5253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:07.943245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50821 (local address 127.0.97.62:36885)
0620 23:36:07.943662 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:07.943667 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:07.945102 (+ 1435us) server_negotiation.cc:407] Connection header received
0620 23:36:07.947454 (+ 2352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:07.947461 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:07.947542 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:07.947650 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:07.949095 (+ 1445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:07.949099 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:07.949103 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:36:07.949117 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:07.949183 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:36:07.949198 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:07.949306 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:07.949846 (+ 540us) server_negotiation.cc:299] Negotiation successful
0620 23:36:07.949905 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":211,"threads_started":1}
I20260620 23:36:07.950608 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998525501163-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998525501163-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998525501163-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36885
--builtin_ntp_servers=127.0.97.20:37559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 23:36:07.951498 5254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:07.943927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36885 (local address 127.0.97.1:50821)
0620 23:36:07.944929 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:07.944972 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:07.947101 (+ 2129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:07.947935 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:07.947948 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:07.948077 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:07.948764 (+ 687us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:36:07.948767 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:07.948815 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:36:07.948923 (+ 108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:36:07.949587 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:36:07.949597 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:36:07.949606 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 23:36:07.949903 (+ 297us) client_negotiation.cc:240] Negotiation successful
0620 23:36:07.950187 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":359,"threads_started":1}
I20260620 23:36:07.953040 5251 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36885
I20260620 23:36:07.953357 5251 heartbeater.cc:464] Registering TS with master...
I20260620 23:36:07.954053 5251 heartbeater.cc:511] Master 127.0.97.62:36885 requested a full tablet report, sending...
I20260620 23:36:07.956570 5065 ts_manager.cc:194] Registered new tserver with Master: 69a230496aab4d7dbd37c889de949355 (127.0.97.1:36023)
W20260620 23:36:08.276549 5255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:36:08.276836 5255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:36:08.276990 5255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:36:08.284516 5255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:36:08.284631 5255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:36:08.284802 5255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 23:36:08.294131 5255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998525501163-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998525501163-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998525501163-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:36:08.295599 5255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:36:08.297434 5255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:36:08.311484 5261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:08.311583 5260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:08.311549 5263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:36:08.312456 5255 server_base.cc:1034] running on GCE node
I20260620 23:36:08.313081 5255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:36:08.314033 5255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:36:08.315255 5255 hybrid_clock.cc:648] HybridClock initialized: now 1781998568315212 us; error 35 us; skew 500 ppm
I20260620 23:36:08.315573 5255 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:36:08.318014 5255 webserver.cc:460] Webserver started at http://127.0.97.2:34555/ using document root <none> and password file <none>
I20260620 23:36:08.318725 5255 fs_manager.cc:359] Metadata directory not provided
I20260620 23:36:08.318818 5255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:36:08.319171 5255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:36:08.322381 5255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998525501163-388-0/minicluster-data/ts-1/data/instance:
uuid: "d01750227bb846e09ec07026ec90286c"
format_stamp: "Formatted at 2026-06-20 23:36:08 on dist-test-slave-tcgx"
server_key: "9bf427185381db62ba1e51f1be49b2fc"
server_key_iv: "e0060015cb1890cb6d9965a82e305b21"
server_key_version: "encryptionkey@0"
I20260620 23:36:08.323238 5255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998525501163-388-0/minicluster-data/ts-1/wal/instance:
uuid: "d01750227bb846e09ec07026ec90286c"
format_stamp: "Formatted at 2026-06-20 23:36:08 on dist-test-slave-tcgx"
server_key: "9bf427185381db62ba1e51f1be49b2fc"
server_key_iv: "e0060015cb1890cb6d9965a82e305b21"
server_key_version: "encryptionkey@0"
I20260620 23:36:08.328886 5255 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:36:08.332811 5269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:08.334398 5255 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 23:36:08.334582 5255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998525501163-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998525501163-388-0/minicluster-data/ts-1/wal
uuid: "d01750227bb846e09ec07026ec90286c"
format_stamp: "Formatted at 2026-06-20 23:36:08 on dist-test-slave-tcgx"
server_key: "9bf427185381db62ba1e51f1be49b2fc"
server_key_iv: "e0060015cb1890cb6d9965a82e305b21"
server_key_version: "encryptionkey@0"
I20260620 23:36:08.334800 5255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998525501163-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998525501163-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998525501163-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:36:08.340941 5255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:36:08.341841 5255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:36:08.342079 5255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:36:08.343032 5255 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:36:08.344421 5255 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:36:08.344511 5255 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:08.344594 5255 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:36:08.344645 5255 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:08.380623 5255 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39971
I20260620 23:36:08.380738 5381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39971 every 8 connection(s)
I20260620 23:36:08.382078 5255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
I20260620 23:36:08.382557 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 5255
I20260620 23:36:08.382761 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998525501163-388-0/minicluster-data/ts-1/wal/instance
I20260620 23:36:08.383425 388 external_mini_cluster.cc:1428] Setting key b1de0d3279abf14890347bdb946398d6
I20260620 23:36:08.386808 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998525501163-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998525501163-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998525501163-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36885
--builtin_ntp_servers=127.0.97.20:37559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 23:36:08.391611 5253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:08.385116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59529 (local address 127.0.97.62:36885)
0620 23:36:08.385325 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:08.385336 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:08.387301 (+ 1965us) server_negotiation.cc:407] Connection header received
0620 23:36:08.389053 (+ 1752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:08.389059 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:08.389130 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:08.389253 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:08.390723 (+ 1470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:08.390732 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:08.390741 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:36:08.390761 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:08.390794 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:36:08.390805 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:08.390947 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:08.391181 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 23:36:08.391255 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 23:36:08.392938 5384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:08.385904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36885 (local address 127.0.97.2:59529)
0620 23:36:08.387103 (+ 1199us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:08.387164 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:08.388754 (+ 1590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:08.389500 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:08.389509 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:08.389618 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:08.390395 (+ 777us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:36:08.390398 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:08.390439 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:36:08.390532 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:36:08.390904 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:36:08.390915 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:36:08.390925 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 23:36:08.391270 (+ 345us) client_negotiation.cc:240] Negotiation successful
0620 23:36:08.391558 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":368,"threads_started":1}
I20260620 23:36:08.394531 5382 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36885
I20260620 23:36:08.394865 5382 heartbeater.cc:464] Registering TS with master...
I20260620 23:36:08.395648 5382 heartbeater.cc:511] Master 127.0.97.62:36885 requested a full tablet report, sending...
I20260620 23:36:08.397570 5065 ts_manager.cc:194] Registered new tserver with Master: d01750227bb846e09ec07026ec90286c (127.0.97.2:39971)
W20260620 23:36:08.727928 5385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:36:08.728222 5385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:36:08.728397 5385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:36:08.735630 5385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:36:08.735733 5385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:36:08.735917 5385 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 23:36:08.745481 5385 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998525501163-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998525501163-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998525501163-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:36:08.746796 5385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:36:08.748621 5385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:36:08.761751 5391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:08.761777 5393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:08.761911 5390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:36:08.763208 5385 server_base.cc:1034] running on GCE node
I20260620 23:36:08.763921 5385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:36:08.764885 5385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:36:08.766098 5385 hybrid_clock.cc:648] HybridClock initialized: now 1781998568766053 us; error 35 us; skew 500 ppm
I20260620 23:36:08.766449 5385 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:36:08.768754 5385 webserver.cc:460] Webserver started at http://127.0.97.3:37489/ using document root <none> and password file <none>
I20260620 23:36:08.769461 5385 fs_manager.cc:359] Metadata directory not provided
I20260620 23:36:08.769551 5385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:36:08.769907 5385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:36:08.772861 5385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998525501163-388-0/minicluster-data/ts-2/data/instance:
uuid: "1fd7e84e009c42c8a14fcb3522a57b47"
format_stamp: "Formatted at 2026-06-20 23:36:08 on dist-test-slave-tcgx"
server_key: "86bb37140318c89f1d79d2ef126acede"
server_key_iv: "6e53674cc5c8a81958b29aee24b26d5b"
server_key_version: "encryptionkey@0"
I20260620 23:36:08.773608 5385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998525501163-388-0/minicluster-data/ts-2/wal/instance:
uuid: "1fd7e84e009c42c8a14fcb3522a57b47"
format_stamp: "Formatted at 2026-06-20 23:36:08 on dist-test-slave-tcgx"
server_key: "86bb37140318c89f1d79d2ef126acede"
server_key_iv: "6e53674cc5c8a81958b29aee24b26d5b"
server_key_version: "encryptionkey@0"
I20260620 23:36:08.778861 5385 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 23:36:08.782590 5399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:08.784118 5385 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:36:08.784353 5385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998525501163-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998525501163-388-0/minicluster-data/ts-2/wal
uuid: "1fd7e84e009c42c8a14fcb3522a57b47"
format_stamp: "Formatted at 2026-06-20 23:36:08 on dist-test-slave-tcgx"
server_key: "86bb37140318c89f1d79d2ef126acede"
server_key_iv: "6e53674cc5c8a81958b29aee24b26d5b"
server_key_version: "encryptionkey@0"
I20260620 23:36:08.784610 5385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998525501163-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998525501163-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998525501163-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:36:08.790540 5385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:36:08.791443 5385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:36:08.791725 5385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:36:08.792698 5385 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:36:08.794134 5385 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:36:08.794255 5385 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:08.794402 5385 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:36:08.794490 5385 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:08.828025 5385 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39825
I20260620 23:36:08.828085 5511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39825 every 8 connection(s)
I20260620 23:36:08.829404 5385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
I20260620 23:36:08.831089 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 5385
I20260620 23:36:08.831290 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998525501163-388-0/minicluster-data/ts-2/wal/instance
I20260620 23:36:08.831807 388 external_mini_cluster.cc:1428] Setting key ac911d3e2932e2b53753f8c53840e4f4
I20260620 23:36:08.838457 5253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:08.832387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44111 (local address 127.0.97.62:36885)
0620 23:36:08.832610 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:08.832617 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:08.834430 (+ 1813us) server_negotiation.cc:407] Connection header received
0620 23:36:08.836383 (+ 1953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:08.836389 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:08.836447 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:08.836541 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:08.837908 (+ 1367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:08.837912 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:08.837916 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:36:08.837930 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:08.837959 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:36:08.837970 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:08.838072 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:08.838202 (+ 130us) server_negotiation.cc:299] Negotiation successful
0620 23:36:08.838248 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 23:36:08.839669 5514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:08.833072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36885 (local address 127.0.97.3:44111)
0620 23:36:08.834252 (+ 1180us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:08.834309 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:08.836062 (+ 1753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:08.836762 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:08.836771 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:08.836871 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:08.837612 (+ 741us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:36:08.837614 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:08.837655 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:36:08.837744 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:36:08.838039 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:36:08.838044 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:36:08.838049 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 23:36:08.838254 (+ 205us) client_negotiation.cc:240] Negotiation successful
0620 23:36:08.838471 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":368,"threads_started":1}
I20260620 23:36:08.841046 5512 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36885
I20260620 23:36:08.841347 5512 heartbeater.cc:464] Registering TS with master...
I20260620 23:36:08.841969 5512 heartbeater.cc:511] Master 127.0.97.62:36885 requested a full tablet report, sending...
I20260620 23:36:08.843437 5065 ts_manager.cc:194] Registered new tserver with Master: 1fd7e84e009c42c8a14fcb3522a57b47 (127.0.97.3:39825)
I20260620 23:36:08.847529 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:36:08.854415 5253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:08.851890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50292 (local address 127.0.97.62:36885)
0620 23:36:08.852071 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:08.852076 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:08.852558 (+ 482us) server_negotiation.cc:407] Connection header received
0620 23:36:08.852693 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:08.852696 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:08.852746 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:08.852830 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:08.853809 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:08.853812 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:08.853816 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:36:08.853829 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:08.853858 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:36:08.853870 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:08.853969 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:08.854104 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 23:36:08.854153 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 23:36:08.858534 5065 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50292:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:36:08.865603 5065 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:36:08.883308 5524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:08.877711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36023 (local address 127.0.0.1:51108)
0620 23:36:08.878688 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:08.878734 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:08.878887 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:08.879635 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:08.879642 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:08.879737 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:08.880600 (+ 863us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:36:08.880606 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:08.880644 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:36:08.880749 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:36:08.882333 (+ 1584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:36:08.882346 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:36:08.882357 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 23:36:08.882562 (+ 205us) client_negotiation.cc:240] Negotiation successful
0620 23:36:08.882670 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":734,"thread_start_us":1698,"threads_started":1}
I20260620 23:36:08.884742 5525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:08.877924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51108 (local address 127.0.97.1:36023)
0620 23:36:08.878699 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:08.878705 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:08.878841 (+ 136us) server_negotiation.cc:407] Connection header received
0620 23:36:08.879138 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:08.879149 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:08.879410 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:08.880211 (+ 801us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:08.882016 (+ 1805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:08.882029 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:08.882049 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:36:08.882078 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:08.882147 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:36:08.882172 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:08.883889 (+ 1717us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:08.883987 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 23:36:08.884245 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":508,"threads_started":1}
I20260620 23:36:08.886039 5527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:08.879075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39971 (local address 127.0.0.1:36326)
0620 23:36:08.879718 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:08.879735 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:08.879880 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:08.883385 (+ 3505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:08.883393 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:08.883429 (+ 36us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:08.884345 (+ 916us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:36:08.884352 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:08.884375 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:36:08.884436 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:36:08.885204 (+ 768us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:36:08.885212 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:36:08.885220 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 23:36:08.885406 (+ 186us) client_negotiation.cc:240] Negotiation successful
0620 23:36:08.885473 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":1297,"threads_started":1}
I20260620 23:36:08.886052 5526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:08.878768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36326 (local address 127.0.97.2:39971)
0620 23:36:08.879334 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:08.879342 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:08.879833 (+ 491us) server_negotiation.cc:407] Connection header received
0620 23:36:08.880020 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:08.880028 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:08.880640 (+ 612us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:08.880807 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:08.884651 (+ 3844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:08.884664 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:08.884685 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:36:08.884723 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:08.884800 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:36:08.884822 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:08.885094 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:08.885419 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 23:36:08.885547 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":477,"threads_started":1}
I20260620 23:36:08.891292 5524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:08.880196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39825 (local address 127.0.0.1:51486)
0620 23:36:08.883545 (+ 3349us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:08.883562 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:08.883639 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:08.889136 (+ 5497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:08.889143 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:08.889180 (+ 37us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:08.889724 (+ 544us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:36:08.889727 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:08.889745 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:36:08.889793 (+ 48us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:36:08.890855 (+ 1062us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:36:08.890860 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:36:08.890864 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:36:08.890973 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 23:36:08.891027 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3260,"thread_start_us":7747,"threads_started":1}
I20260620 23:36:08.890857 5317 tablet_service.cc:1467] Processing CreateTablet for tablet 21ac1c0206194613b11c1fe7b47f6b0c (DEFAULT_TABLE table=test-table [id=a7614643f5814906822d1f75c38f425e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:36:08.892122 5529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:08.880508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51486 (local address 127.0.97.3:39825)
0620 23:36:08.888593 (+ 8085us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:08.888601 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:08.888633 (+ 32us) server_negotiation.cc:407] Connection header received
0620 23:36:08.888696 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:08.888703 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:08.888944 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:08.889507 (+ 563us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:08.890601 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:08.890607 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:08.890616 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:36:08.890626 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:08.890699 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:36:08.890717 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:08.891565 (+ 848us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:08.891628 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 23:36:08.891717 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7925,"thread_start_us":7714,"threads_started":1}
I20260620 23:36:08.891618 5186 tablet_service.cc:1467] Processing CreateTablet for tablet 21ac1c0206194613b11c1fe7b47f6b0c (DEFAULT_TABLE table=test-table [id=a7614643f5814906822d1f75c38f425e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:36:08.892426 5317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21ac1c0206194613b11c1fe7b47f6b0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:36:08.892961 5186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21ac1c0206194613b11c1fe7b47f6b0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:36:08.895062 5447 tablet_service.cc:1467] Processing CreateTablet for tablet 21ac1c0206194613b11c1fe7b47f6b0c (DEFAULT_TABLE table=test-table [id=a7614643f5814906822d1f75c38f425e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:36:08.896327 5447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21ac1c0206194613b11c1fe7b47f6b0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:36:08.907982 5532 tablet_bootstrap.cc:492] T 21ac1c0206194613b11c1fe7b47f6b0c P d01750227bb846e09ec07026ec90286c: Bootstrap starting.
I20260620 23:36:08.908864 5530 tablet_bootstrap.cc:492] T 21ac1c0206194613b11c1fe7b47f6b0c P 1fd7e84e009c42c8a14fcb3522a57b47: Bootstrap starting.
I20260620 23:36:08.911003 5531 tablet_bootstrap.cc:492] T 21ac1c0206194613b11c1fe7b47f6b0c P 69a230496aab4d7dbd37c889de949355: Bootstrap starting.
I20260620 23:36:08.913208 5530 tablet_bootstrap.cc:654] T 21ac1c0206194613b11c1fe7b47f6b0c P 1fd7e84e009c42c8a14fcb3522a57b47: Neither blocks nor log segments found. Creating new log.
I20260620 23:36:08.914050 5532 tablet_bootstrap.cc:654] T 21ac1c0206194613b11c1fe7b47f6b0c P d01750227bb846e09ec07026ec90286c: Neither blocks nor log segments found. Creating new log.
I20260620 23:36:08.914515 5531 tablet_bootstrap.cc:654] T 21ac1c0206194613b11c1fe7b47f6b0c P 69a230496aab4d7dbd37c889de949355: Neither blocks nor log segments found. Creating new log.
I20260620 23:36:08.914529 5530 log.cc:824] T 21ac1c0206194613b11c1fe7b47f6b0c P 1fd7e84e009c42c8a14fcb3522a57b47: Log is configured to *not* fsync() on all Append() calls
I20260620 23:36:08.915126 5532 log.cc:824] T 21ac1c0206194613b11c1fe7b47f6b0c P d01750227bb846e09ec07026ec90286c: Log is configured to *not* fsync() on all Append() calls
I20260620 23:36:08.915620 5531 log.cc:824] T 21ac1c0206194613b11c1fe7b47f6b0c P 69a230496aab4d7dbd37c889de949355: Log is configured to *not* fsync() on all Append() calls
I20260620 23:36:08.917671 5530 tablet_bootstrap.cc:492] T 21ac1c0206194613b11c1fe7b47f6b0c P 1fd7e84e009c42c8a14fcb3522a57b47: No bootstrap required, opened a new log
I20260620 23:36:08.917954 5530 ts_tablet_manager.cc:1397] T 21ac1c0206194613b11c1fe7b47f6b0c P 1fd7e84e009c42c8a14fcb3522a57b47: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260620 23:36:08.918080 5532 tablet_bootstrap.cc:492] T 21ac1c0206194613b11c1fe7b47f6b0c P d01750227bb846e09ec07026ec90286c: No bootstrap required, opened a new log
I20260620 23:36:08.918334 5532 ts_tablet_manager.cc:1397] T 21ac1c0206194613b11c1fe7b47f6b0c P d01750227bb846e09ec07026ec90286c: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20260620 23:36:08.918483 5531 tablet_bootstrap.cc:492] T 21ac1c0206194613b11c1fe7b47f6b0c P 69a230496aab4d7dbd37c889de949355: No bootstrap required, opened a new log
I20260620 23:36:08.918836 5531 ts_tablet_manager.cc:1397] T 21ac1c0206194613b11c1fe7b47f6b0c P 69a230496aab4d7dbd37c889de949355: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260620 23:36:08.925518 5530 raft_consensus.cc:348] T 21ac1c0206194613b11c1fe7b47f6b0c P 1fd7e84e009c42c8a14fcb3522a57b47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a230496aab4d7dbd37c889de949355" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36023 } } peers { permanent_uuid: "1fd7e84e009c42c8a14fcb3522a57b47" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39825 } } peers { permanent_uuid: "d01750227bb846e09ec07026ec90286c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39971 } }
I20260620 23:36:08.925916 5530 raft_consensus.cc:374] T 21ac1c0206194613b11c1fe7b47f6b0c P 1fd7e84e009c42c8a14fcb3522a57b47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:36:08.926055 5530 raft_consensus.cc:729] T 21ac1c0206194613b11c1fe7b47f6b0c P 1fd7e84e009c42c8a14fcb3522a57b47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fd7e84e009c42c8a14fcb3522a57b47, State: Initialized, Role: FOLLOWER
I20260620 23:36:08.926697 5530 consensus_queue.cc:260] T 21ac1c0206194613b11c1fe7b47f6b0c P 1fd7e84e009c42c8a14fcb3522a57b47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a230496aab4d7dbd37c889de949355" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36023 } } peers { permanent_uuid: "1fd7e84e009c42c8a14fcb3522a57b47" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39825 } } peers { permanent_uuid: "d01750227bb846e09ec07026ec90286c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39971 } }
I20260620 23:36:08.927845 5512 heartbeater.cc:503] Master 127.0.97.62:36885 was elected leader, sending a full tablet report...
I20260620 23:36:08.929100 5531 raft_consensus.cc:348] T 21ac1c0206194613b11c1fe7b47f6b0c P 69a230496aab4d7dbd37c889de949355 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a230496aab4d7dbd37c889de949355" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36023 } } peers { permanent_uuid: "1fd7e84e009c42c8a14fcb3522a57b47" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39825 } } peers { permanent_uuid: "d01750227bb846e09ec07026ec90286c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39971 } }
I20260620 23:36:08.929045 5532 raft_consensus.cc:348] T 21ac1c0206194613b11c1fe7b47f6b0c P d01750227bb846e09ec07026ec90286c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a230496aab4d7dbd37c889de949355" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36023 } } peers { permanent_uuid: "1fd7e84e009c42c8a14fcb3522a57b47" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39825 } } peers { permanent_uuid: "d01750227bb846e09ec07026ec90286c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39971 } }
I20260620 23:36:08.929490 5532 raft_consensus.cc:374] T 21ac1c0206194613b11c1fe7b47f6b0c P d01750227bb846e09ec07026ec90286c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:36:08.929565 5531 raft_consensus.cc:374] T 21ac1c0206194613b11c1fe7b47f6b0c P 69a230496aab4d7dbd37c889de949355 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:36:08.929632 5532 raft_consensus.cc:729] T 21ac1c0206194613b11c1fe7b47f6b0c P d01750227bb846e09ec07026ec90286c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d01750227bb846e09ec07026ec90286c, State: Initialized, Role: FOLLOWER
I20260620 23:36:08.929699 5531 raft_consensus.cc:729] T 21ac1c0206194613b11c1fe7b47f6b0c P 69a230496aab4d7dbd37c889de949355 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69a230496aab4d7dbd37c889de949355, State: Initialized, Role: FOLLOWER
I20260620 23:36:08.929844 5530 ts_tablet_manager.cc:1428] T 21ac1c0206194613b11c1fe7b47f6b0c P 1fd7e84e009c42c8a14fcb3522a57b47: Time spent starting tablet: real 0.012s user 0.011s sys 0.001s
I20260620 23:36:08.930430 5532 consensus_queue.cc:260] T 21ac1c0206194613b11c1fe7b47f6b0c P d01750227bb846e09ec07026ec90286c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a230496aab4d7dbd37c889de949355" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36023 } } peers { permanent_uuid: "1fd7e84e009c42c8a14fcb3522a57b47" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39825 } } peers { permanent_uuid: "d01750227bb846e09ec07026ec90286c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39971 } }
I20260620 23:36:08.930430 5531 consensus_queue.cc:260] T 21ac1c0206194613b11c1fe7b47f6b0c P 69a230496aab4d7dbd37c889de949355 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a230496aab4d7dbd37c889de949355" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36023 } } peers { permanent_uuid: "1fd7e84e009c42c8a14fcb3522a57b47" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39825 } } peers { permanent_uuid: "d01750227bb846e09ec07026ec90286c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39971 } }
I20260620 23:36:08.931778 5251 heartbeater.cc:503] Master 127.0.97.62:36885 was elected leader, sending a full tablet report...
I20260620 23:36:08.932230 5382 heartbeater.cc:503] Master 127.0.97.62:36885 was elected leader, sending a full tablet report...
I20260620 23:36:08.932473 5531 ts_tablet_manager.cc:1428] T 21ac1c0206194613b11c1fe7b47f6b0c P 69a230496aab4d7dbd37c889de949355: Time spent starting tablet: real 0.013s user 0.011s sys 0.001s
I20260620 23:36:08.933784 5532 ts_tablet_manager.cc:1428] T 21ac1c0206194613b11c1fe7b47f6b0c P d01750227bb846e09ec07026ec90286c: Time spent starting tablet: real 0.015s user 0.008s sys 0.005s
W20260620 23:36:08.942884 5252 tablet.cc:2369] T 21ac1c0206194613b11c1fe7b47f6b0c P 69a230496aab4d7dbd37c889de949355: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:36:09.065177 5538 raft_consensus.cc:482] T 21ac1c0206194613b11c1fe7b47f6b0c P d01750227bb846e09ec07026ec90286c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:36:09.065485 5538 raft_consensus.cc:504] T 21ac1c0206194613b11c1fe7b47f6b0c P d01750227bb846e09ec07026ec90286c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a230496aab4d7dbd37c889de949355" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36023 } } peers { permanent_uuid: "1fd7e84e009c42c8a14fcb3522a57b47" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39825 } } peers { permanent_uuid: "d01750227bb846e09ec07026ec90286c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39971 } }
I20260620 23:36:09.067073 5538 leader_election.cc:288] T 21ac1c0206194613b11c1fe7b47f6b0c P d01750227bb846e09ec07026ec90286c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69a230496aab4d7dbd37c889de949355 (127.0.97.1:36023), 1fd7e84e009c42c8a14fcb3522a57b47 (127.0.97.3:39825)
I20260620 23:36:09.070698 5525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:09.067495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39695 (local address 127.0.97.1:36023)
0620 23:36:09.067695 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:09.067702 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:09.068381 (+ 679us) server_negotiation.cc:407] Connection header received
0620 23:36:09.068494 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:09.068502 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:09.068590 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:09.068711 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:09.069990 (+ 1279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:09.069994 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:09.069997 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:36:09.070011 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:09.070063 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:36:09.070079 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:09.070194 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:09.070356 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 23:36:09.070434 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 23:36:09.070843 5539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:09.067496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36023 (local address 127.0.97.2:39695)
0620 23:36:09.068164 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:09.068180 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:09.068357 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:09.068876 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:09.068883 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:09.068918 (+ 35us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:09.069741 (+ 823us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:36:09.069746 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:09.069770 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:36:09.069825 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:36:09.070174 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:36:09.070177 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:36:09.070181 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:36:09.070377 (+ 196us) client_negotiation.cc:240] Negotiation successful
0620 23:36:09.070447 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":389,"threads_started":1}
I20260620 23:36:09.071440 5529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:09.068536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44097 (local address 127.0.97.3:39825)
0620 23:36:09.068715 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:09.068720 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:09.069369 (+ 649us) server_negotiation.cc:407] Connection header received
0620 23:36:09.069457 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:09.069463 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:09.069521 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:09.069618 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:09.070637 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:09.070643 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:09.070647 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:36:09.070664 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:09.070703 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:36:09.070718 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:09.070814 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:09.071112 (+ 298us) server_negotiation.cc:299] Negotiation successful
0620 23:36:09.071172 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 23:36:09.071655 5540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:09.068254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39825 (local address 127.0.97.2:44097)
0620 23:36:09.069241 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:09.069259 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:09.069362 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:09.069678 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:09.069684 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:09.069713 (+ 29us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:09.070408 (+ 695us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:36:09.070411 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:09.070424 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:36:09.070460 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:36:09.070903 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:36:09.070910 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:36:09.070916 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 23:36:09.071129 (+ 213us) client_negotiation.cc:240] Negotiation successful
0620 23:36:09.071172 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":855,"thread_start_us":599,"threads_started":1}
I20260620 23:36:09.071966 5206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21ac1c0206194613b11c1fe7b47f6b0c" candidate_uuid: "d01750227bb846e09ec07026ec90286c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69a230496aab4d7dbd37c889de949355" is_pre_election: true
I20260620 23:36:09.072562 5206 raft_consensus.cc:2454] T 21ac1c0206194613b11c1fe7b47f6b0c P 69a230496aab4d7dbd37c889de949355 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d01750227bb846e09ec07026ec90286c in term 0.
I20260620 23:36:09.072711 5467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21ac1c0206194613b11c1fe7b47f6b0c" candidate_uuid: "d01750227bb846e09ec07026ec90286c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fd7e84e009c42c8a14fcb3522a57b47" is_pre_election: true
I20260620 23:36:09.073210 5467 raft_consensus.cc:2454] T 21ac1c0206194613b11c1fe7b47f6b0c P 1fd7e84e009c42c8a14fcb3522a57b47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d01750227bb846e09ec07026ec90286c in term 0.
I20260620 23:36:09.073267 5273 leader_election.cc:302] T 21ac1c0206194613b11c1fe7b47f6b0c P d01750227bb846e09ec07026ec90286c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69a230496aab4d7dbd37c889de949355, d01750227bb846e09ec07026ec90286c; no voters:
I20260620 23:36:09.073712 5538 raft_consensus.cc:2781] T 21ac1c0206194613b11c1fe7b47f6b0c P d01750227bb846e09ec07026ec90286c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:36:09.073861 5538 raft_consensus.cc:482] T 21ac1c0206194613b11c1fe7b47f6b0c P d01750227bb846e09ec07026ec90286c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:36:09.073959 5538 raft_consensus.cc:3037] T 21ac1c0206194613b11c1fe7b47f6b0c P d01750227bb846e09ec07026ec90286c [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:36:09.076164 5538 raft_consensus.cc:504] T 21ac1c0206194613b11c1fe7b47f6b0c P d01750227bb846e09ec07026ec90286c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a230496aab4d7dbd37c889de949355" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36023 } } peers { permanent_uuid: "1fd7e84e009c42c8a14fcb3522a57b47" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39825 } } peers { permanent_uuid: "d01750227bb846e09ec07026ec90286c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39971 } }
I20260620 23:36:09.076678 5538 leader_election.cc:288] T 21ac1c0206194613b11c1fe7b47f6b0c P d01750227bb846e09ec07026ec90286c [CANDIDATE]: Term 1 election: Requested vote from peers 69a230496aab4d7dbd37c889de949355 (127.0.97.1:36023), 1fd7e84e009c42c8a14fcb3522a57b47 (127.0.97.3:39825)
I20260620 23:36:09.077088 5206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21ac1c0206194613b11c1fe7b47f6b0c" candidate_uuid: "d01750227bb846e09ec07026ec90286c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69a230496aab4d7dbd37c889de949355"
I20260620 23:36:09.077157 5467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21ac1c0206194613b11c1fe7b47f6b0c" candidate_uuid: "d01750227bb846e09ec07026ec90286c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fd7e84e009c42c8a14fcb3522a57b47"
I20260620 23:36:09.077266 5206 raft_consensus.cc:3037] T 21ac1c0206194613b11c1fe7b47f6b0c P 69a230496aab4d7dbd37c889de949355 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:36:09.077379 5467 raft_consensus.cc:3037] T 21ac1c0206194613b11c1fe7b47f6b0c P 1fd7e84e009c42c8a14fcb3522a57b47 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:36:09.079563 5206 raft_consensus.cc:2454] T 21ac1c0206194613b11c1fe7b47f6b0c P 69a230496aab4d7dbd37c889de949355 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d01750227bb846e09ec07026ec90286c in term 1.
I20260620 23:36:09.079623 5467 raft_consensus.cc:2454] T 21ac1c0206194613b11c1fe7b47f6b0c P 1fd7e84e009c42c8a14fcb3522a57b47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d01750227bb846e09ec07026ec90286c in term 1.
I20260620 23:36:09.079996 5273 leader_election.cc:302] T 21ac1c0206194613b11c1fe7b47f6b0c P d01750227bb846e09ec07026ec90286c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69a230496aab4d7dbd37c889de949355, d01750227bb846e09ec07026ec90286c; no voters:
I20260620 23:36:09.080320 5538 raft_consensus.cc:2781] T 21ac1c0206194613b11c1fe7b47f6b0c P d01750227bb846e09ec07026ec90286c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:36:09.080892 5538 raft_consensus.cc:686] T 21ac1c0206194613b11c1fe7b47f6b0c P d01750227bb846e09ec07026ec90286c [term 1 LEADER]: Becoming Leader. State: Replica: d01750227bb846e09ec07026ec90286c, State: Running, Role: LEADER
W20260620 23:36:09.080948 5513 tablet.cc:2369] T 21ac1c0206194613b11c1fe7b47f6b0c P 1fd7e84e009c42c8a14fcb3522a57b47: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:36:09.081380 5538 consensus_queue.cc:237] T 21ac1c0206194613b11c1fe7b47f6b0c P d01750227bb846e09ec07026ec90286c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a230496aab4d7dbd37c889de949355" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36023 } } peers { permanent_uuid: "1fd7e84e009c42c8a14fcb3522a57b47" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39825 } } peers { permanent_uuid: "d01750227bb846e09ec07026ec90286c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39971 } }
I20260620 23:36:09.085922 5064 catalog_manager.cc:5495] T 21ac1c0206194613b11c1fe7b47f6b0c P d01750227bb846e09ec07026ec90286c reported cstate change: term changed from 0 to 1, leader changed from <none> to d01750227bb846e09ec07026ec90286c (127.0.97.2). New cstate: current_term: 1 leader_uuid: "d01750227bb846e09ec07026ec90286c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a230496aab4d7dbd37c889de949355" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fd7e84e009c42c8a14fcb3522a57b47" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d01750227bb846e09ec07026ec90286c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39971 } health_report { overall_health: HEALTHY } } }
I20260620 23:36:09.127058 5526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:09.124906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36334 (local address 127.0.97.2:39971)
0620 23:36:09.125133 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:09.125138 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:09.125209 (+ 71us) server_negotiation.cc:407] Connection header received
0620 23:36:09.125434 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:09.125438 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:09.125514 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:09.125605 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:09.126474 (+ 869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:09.126478 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:09.126481 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:36:09.126496 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:09.126524 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:36:09.126535 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:09.126630 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:09.126710 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 23:36:09.126752 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
W20260620 23:36:09.134496 5383 tablet.cc:2369] T 21ac1c0206194613b11c1fe7b47f6b0c P d01750227bb846e09ec07026ec90286c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:36:09.135438 5467 raft_consensus.cc:1261] T 21ac1c0206194613b11c1fe7b47f6b0c P 1fd7e84e009c42c8a14fcb3522a57b47 [term 1 FOLLOWER]: Refusing update from remote peer d01750227bb846e09ec07026ec90286c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:36:09.135457 5206 raft_consensus.cc:1261] T 21ac1c0206194613b11c1fe7b47f6b0c P 69a230496aab4d7dbd37c889de949355 [term 1 FOLLOWER]: Refusing update from remote peer d01750227bb846e09ec07026ec90286c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:36:09.136418 5541 consensus_queue.cc:1035] T 21ac1c0206194613b11c1fe7b47f6b0c P d01750227bb846e09ec07026ec90286c [LEADER]: Connected to new peer: Peer: permanent_uuid: "69a230496aab4d7dbd37c889de949355" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:36:09.136855 5538 consensus_queue.cc:1035] T 21ac1c0206194613b11c1fe7b47f6b0c P d01750227bb846e09ec07026ec90286c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fd7e84e009c42c8a14fcb3522a57b47" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:36:09.162395 5063 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:36:09.162696 5063 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:36:09.166379 5063 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ac882235da6b4bb29c5f469d9415fe09: Sending DeleteTablet for 3 replicas of tablet 21ac1c0206194613b11c1fe7b47f6b0c
I20260620 23:36:09.167680 5186 tablet_service.cc:1514] Processing DeleteTablet for tablet 21ac1c0206194613b11c1fe7b47f6b0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:36:09 UTC) from {username='slave'} at 127.0.0.1:51108
I20260620 23:36:09.167652 5317 tablet_service.cc:1514] Processing DeleteTablet for tablet 21ac1c0206194613b11c1fe7b47f6b0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:36:09 UTC) from {username='slave'} at 127.0.0.1:36326
I20260620 23:36:09.168093 5447 tablet_service.cc:1514] Processing DeleteTablet for tablet 21ac1c0206194613b11c1fe7b47f6b0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:36:09 UTC) from {username='slave'} at 127.0.0.1:51486
I20260620 23:36:09.168959 5556 tablet_replica.cc:330] T 21ac1c0206194613b11c1fe7b47f6b0c P 1fd7e84e009c42c8a14fcb3522a57b47: stopping tablet replica
I20260620 23:36:09.169513 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 5124
I20260620 23:36:09.170683 5556 raft_consensus.cc:2229] T 21ac1c0206194613b11c1fe7b47f6b0c P 1fd7e84e009c42c8a14fcb3522a57b47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:36:09.171252 5556 raft_consensus.cc:2258] T 21ac1c0206194613b11c1fe7b47f6b0c P 1fd7e84e009c42c8a14fcb3522a57b47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:36:09.173658 5556 ts_tablet_manager.cc:1905] T 21ac1c0206194613b11c1fe7b47f6b0c P 1fd7e84e009c42c8a14fcb3522a57b47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:36:09.182750 5556 ts_tablet_manager.cc:1918] T 21ac1c0206194613b11c1fe7b47f6b0c P 1fd7e84e009c42c8a14fcb3522a57b47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:36:09.183027 5556 log.cc:1196] T 21ac1c0206194613b11c1fe7b47f6b0c P 1fd7e84e009c42c8a14fcb3522a57b47: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998525501163-388-0/minicluster-data/ts-2/wal/wals/21ac1c0206194613b11c1fe7b47f6b0c
I20260620 23:36:09.183617 5556 ts_tablet_manager.cc:1939] T 21ac1c0206194613b11c1fe7b47f6b0c P 1fd7e84e009c42c8a14fcb3522a57b47: Deleting consensus metadata
I20260620 23:36:09.184597 5525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:09.171506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51114 (local address 127.0.97.1:36023)
0620 23:36:09.171713 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:09.171719 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:09.181491 (+ 9772us) server_negotiation.cc:407] Connection header received
0620 23:36:09.181979 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:09.181985 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:09.182109 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:09.182184 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:09.183689 (+ 1505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:09.183697 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:09.183703 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:36:09.183723 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:09.183759 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:36:09.183778 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:09.183887 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:09.184185 (+ 298us) server_negotiation.cc:299] Negotiation successful
0620 23:36:09.184252 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 23:36:09.185632 5052 catalog_manager.cc:4841] TS 1fd7e84e009c42c8a14fcb3522a57b47 (127.0.97.3:39825): tablet 21ac1c0206194613b11c1fe7b47f6b0c (table test-table [id=a7614643f5814906822d1f75c38f425e]) successfully deleted
I20260620 23:36:09.185818 5246 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:36:09.188004 5555 tablet_replica.cc:330] T 21ac1c0206194613b11c1fe7b47f6b0c P d01750227bb846e09ec07026ec90286c: stopping tablet replica
I20260620 23:36:09.189316 5555 raft_consensus.cc:2229] T 21ac1c0206194613b11c1fe7b47f6b0c P d01750227bb846e09ec07026ec90286c [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:36:09.190344 5555 raft_consensus.cc:2258] T 21ac1c0206194613b11c1fe7b47f6b0c P d01750227bb846e09ec07026ec90286c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:36:09.193114 5555 ts_tablet_manager.cc:1905] T 21ac1c0206194613b11c1fe7b47f6b0c P d01750227bb846e09ec07026ec90286c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:36:09.199751 5555 ts_tablet_manager.cc:1918] T 21ac1c0206194613b11c1fe7b47f6b0c P d01750227bb846e09ec07026ec90286c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:36:09.200033 5555 log.cc:1196] T 21ac1c0206194613b11c1fe7b47f6b0c P d01750227bb846e09ec07026ec90286c: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998525501163-388-0/minicluster-data/ts-1/wal/wals/21ac1c0206194613b11c1fe7b47f6b0c
I20260620 23:36:09.200546 5555 ts_tablet_manager.cc:1939] T 21ac1c0206194613b11c1fe7b47f6b0c P d01750227bb846e09ec07026ec90286c: Deleting consensus metadata
I20260620 23:36:09.202239 5049 catalog_manager.cc:4841] TS d01750227bb846e09ec07026ec90286c (127.0.97.2:39971): tablet 21ac1c0206194613b11c1fe7b47f6b0c (table test-table [id=a7614643f5814906822d1f75c38f425e]) successfully deleted
I20260620 23:36:09.301081 5554 tablet_replica.cc:330] T 21ac1c0206194613b11c1fe7b47f6b0c P 69a230496aab4d7dbd37c889de949355: stopping tablet replica
I20260620 23:36:09.301704 5554 raft_consensus.cc:2229] T 21ac1c0206194613b11c1fe7b47f6b0c P 69a230496aab4d7dbd37c889de949355 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:36:09.302112 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 5124
I20260620 23:36:09.324499 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 5255
W20260620 23:36:09.324515 5052 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:36023 (error 108)
I20260620 23:36:09.326920 5524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:09.326435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36023 (local address 127.0.0.1:51130)
0620 23:36:09.326633 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:09.326771 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260620 23:36:09.327193 5052 catalog_manager.cc:4584] TS 69a230496aab4d7dbd37c889de949355 (127.0.97.1:36023): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 21ac1c0206194613b11c1fe7b47f6b0c: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36023: connect: Connection refused (error 111)
I20260620 23:36:09.327832 5526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:09.325352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36346 (local address 127.0.97.2:39971)
0620 23:36:09.325528 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:09.325534 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:09.325612 (+ 78us) server_negotiation.cc:407] Connection header received
0620 23:36:09.325838 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:09.325843 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:09.325907 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:09.326013 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:09.327074 (+ 1061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:09.327080 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:09.327084 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:36:09.327104 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:09.327145 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:36:09.327163 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:09.327295 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:09.327528 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 23:36:09.327582 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 23:36:09.328547 5377 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:36:09.381057 5524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:09.380494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36023 (local address 127.0.0.1:51136)
0620 23:36:09.380747 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:09.380868 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260620 23:36:09.382241 5524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:09.381822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36023 (local address 127.0.0.1:51144)
0620 23:36:09.381979 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:09.382077 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 23:36:09.438477 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 5255
I20260620 23:36:09.442770 5524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:09.442339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36023 (local address 127.0.0.1:51154)
0620 23:36:09.442545 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:09.442638 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260620 23:36:09.443935 5524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:09.443566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36023 (local address 127.0.0.1:51156)
0620 23:36:09.443726 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:09.443813 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 23:36:09.456663 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 5385
I20260620 23:36:09.459561 5529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:09.457475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51490 (local address 127.0.97.3:39825)
0620 23:36:09.457668 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:09.457673 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:09.457692 (+ 19us) server_negotiation.cc:407] Connection header received
0620 23:36:09.457844 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:09.457848 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:09.457936 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:09.458022 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:09.458883 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:09.458887 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:09.458891 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:36:09.458904 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:09.458971 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:36:09.458989 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:09.459103 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:09.459177 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 23:36:09.459229 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 23:36:09.460247 5507 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:36:09.524899 5524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:09.524357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36023 (local address 127.0.0.1:51160)
0620 23:36:09.524630 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:09.524721 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
I20260620 23:36:09.526024 5524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:09.525578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36023 (local address 127.0.0.1:51176)
0620 23:36:09.525805 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:09.525877 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
I20260620 23:36:09.561214 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 5385
I20260620 23:36:09.579504 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 5034
I20260620 23:36:09.580674 5095 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:36:09.692375 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 5034
2026-06-20T23:36:09Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2808 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 23:36:09.733729 388 mini_oidc.cc:152] Starting JWKS server
I20260620 23:36:09.736398 388 webserver.cc:460] Webserver started at https://127.0.0.1:45249/ using document root <none> and password file <none>
I20260620 23:36:09.736707 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 23:36:09.737381 388 webserver.cc:460] Webserver started at http://127.0.0.1:42685/ using document root <none> and password file <none>
2026-06-20T23:36:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:36:09Z Disabled control of system clock
I20260620 23:36:09.765631 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781998525501163-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781998525501163-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781998525501163-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42629
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:44099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42629
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45249/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781998525501163-388-0/testchainca.cert with env {}
W20260620 23:36:10.099038 5568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:36:10.099375 5568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:36:10.099464 5568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:36:10.106889 5568 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:36:10.107106 5568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:36:10.107151 5568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:36:10.107208 5568 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:36:10.107254 5568 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 23:36:10.107308 5568 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 23:36:10.107340 5568 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45249/jwks.json
I20260620 23:36:10.117820 5568 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781998525501163-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781998525501163-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42629
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45249/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781998525501163-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781998525501163-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:36:10.119186 5568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:36:10.120463 5568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:36:10.132853 5573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:10.132910 5576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:10.132943 5574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:36:10.134177 5568 server_base.cc:1034] running on GCE node
I20260620 23:36:10.135136 5568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:36:10.136601 5568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:36:10.137813 5568 hybrid_clock.cc:648] HybridClock initialized: now 1781998570137745 us; error 67 us; skew 500 ppm
I20260620 23:36:10.138159 5568 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:36:10.140461 5568 webserver.cc:460] Webserver started at http://127.0.97.62:39671/ using document root <none> and password file <none>
I20260620 23:36:10.141134 5568 fs_manager.cc:359] Metadata directory not provided
I20260620 23:36:10.141222 5568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:36:10.141477 5568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:36:10.144448 5568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781998525501163-388-0/minicluster-data/master-0/data/instance:
uuid: "93162fb61aec41918fc40ebf55fb5a1e"
format_stamp: "Formatted at 2026-06-20 23:36:10 on dist-test-slave-tcgx"
server_key: "8585612a08ff04547a6376c56349394f"
server_key_iv: "34eceaa881db9c87cb60fb561a2f0d1e"
server_key_version: "encryptionkey@0"
I20260620 23:36:10.145262 5568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781998525501163-388-0/minicluster-data/master-0/wal/instance:
uuid: "93162fb61aec41918fc40ebf55fb5a1e"
format_stamp: "Formatted at 2026-06-20 23:36:10 on dist-test-slave-tcgx"
server_key: "8585612a08ff04547a6376c56349394f"
server_key_iv: "34eceaa881db9c87cb60fb561a2f0d1e"
server_key_version: "encryptionkey@0"
I20260620 23:36:10.150477 5568 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 23:36:10.153949 5582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:10.155490 5568 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:36:10.155635 5568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781998525501163-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781998525501163-388-0/minicluster-data/master-0/wal
uuid: "93162fb61aec41918fc40ebf55fb5a1e"
format_stamp: "Formatted at 2026-06-20 23:36:10 on dist-test-slave-tcgx"
server_key: "8585612a08ff04547a6376c56349394f"
server_key_iv: "34eceaa881db9c87cb60fb561a2f0d1e"
server_key_version: "encryptionkey@0"
I20260620 23:36:10.155817 5568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781998525501163-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781998525501163-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781998525501163-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:36:10.173867 5568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:36:10.174678 5568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:36:10.174922 5568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:36:10.193948 5568 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42629
I20260620 23:36:10.193954 5633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42629 every 8 connection(s)
I20260620 23:36:10.195376 5568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:36:10.196117 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 5568
I20260620 23:36:10.196305 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781998525501163-388-0/minicluster-data/master-0/wal/instance
I20260620 23:36:10.197103 388 external_mini_cluster.cc:1428] Setting key afaf4b0022d52e7e50495cef49631365
I20260620 23:36:10.200085 5634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:36:10.213158 5637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:10.199167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37188 (local address 127.0.97.62:42629)
0620 23:36:10.200219 (+ 1052us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:10.200234 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:10.200283 (+ 49us) server_negotiation.cc:407] Connection header received
0620 23:36:10.202546 (+ 2263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:10.202576 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:10.203238 (+ 662us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:10.203703 (+ 465us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:10.204247 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:10.206351 (+ 2104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:10.207791 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:10.208795 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:10.209000 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:10.210174 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:10.210185 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:10.210205 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:36:10.211024 (+ 819us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:10.211123 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:36:10.211155 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:10.211272 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:10.211495 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 23:36:10.211860 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":314,"threads_started":1}
I20260620 23:36:10.214219 5634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93162fb61aec41918fc40ebf55fb5a1e: Bootstrap starting.
I20260620 23:36:10.217568 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:36:10.218150 5634 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93162fb61aec41918fc40ebf55fb5a1e: Neither blocks nor log segments found. Creating new log.
I20260620 23:36:10.219412 5634 log.cc:824] T 00000000000000000000000000000000 P 93162fb61aec41918fc40ebf55fb5a1e: Log is configured to *not* fsync() on all Append() calls
I20260620 23:36:10.222296 5634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93162fb61aec41918fc40ebf55fb5a1e: No bootstrap required, opened a new log
I20260620 23:36:10.229485 5634 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93162fb61aec41918fc40ebf55fb5a1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93162fb61aec41918fc40ebf55fb5a1e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42629 } }
I20260620 23:36:10.229827 5634 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93162fb61aec41918fc40ebf55fb5a1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:36:10.229910 5634 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93162fb61aec41918fc40ebf55fb5a1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93162fb61aec41918fc40ebf55fb5a1e, State: Initialized, Role: FOLLOWER
I20260620 23:36:10.230391 5634 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93162fb61aec41918fc40ebf55fb5a1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93162fb61aec41918fc40ebf55fb5a1e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42629 } }
I20260620 23:36:10.230592 5634 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93162fb61aec41918fc40ebf55fb5a1e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:36:10.230682 5634 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93162fb61aec41918fc40ebf55fb5a1e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:36:10.230803 5634 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93162fb61aec41918fc40ebf55fb5a1e [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:36:10.232900 5634 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93162fb61aec41918fc40ebf55fb5a1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93162fb61aec41918fc40ebf55fb5a1e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42629 } }
I20260620 23:36:10.233480 5634 leader_election.cc:302] T 00000000000000000000000000000000 P 93162fb61aec41918fc40ebf55fb5a1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93162fb61aec41918fc40ebf55fb5a1e; no voters:
I20260620 23:36:10.234179 5634 leader_election.cc:288] T 00000000000000000000000000000000 P 93162fb61aec41918fc40ebf55fb5a1e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:36:10.234340 5639 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93162fb61aec41918fc40ebf55fb5a1e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:36:10.235066 5639 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93162fb61aec41918fc40ebf55fb5a1e [term 1 LEADER]: Becoming Leader. State: Replica: 93162fb61aec41918fc40ebf55fb5a1e, State: Running, Role: LEADER
I20260620 23:36:10.235601 5639 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93162fb61aec41918fc40ebf55fb5a1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93162fb61aec41918fc40ebf55fb5a1e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42629 } }
I20260620 23:36:10.236294 5634 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93162fb61aec41918fc40ebf55fb5a1e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:36:10.238543 5640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93162fb61aec41918fc40ebf55fb5a1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93162fb61aec41918fc40ebf55fb5a1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93162fb61aec41918fc40ebf55fb5a1e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42629 } } }
I20260620 23:36:10.239046 5640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93162fb61aec41918fc40ebf55fb5a1e [sys.catalog]: This master's current role is: LEADER
I20260620 23:36:10.239583 5641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93162fb61aec41918fc40ebf55fb5a1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93162fb61aec41918fc40ebf55fb5a1e. Latest consensus state: current_term: 1 leader_uuid: "93162fb61aec41918fc40ebf55fb5a1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93162fb61aec41918fc40ebf55fb5a1e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42629 } } }
I20260620 23:36:10.239931 5641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93162fb61aec41918fc40ebf55fb5a1e [sys.catalog]: This master's current role is: LEADER
W20260620 23:36:10.246755 5655 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 93162fb61aec41918fc40ebf55fb5a1e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:36:10.246912 5655 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:36:10.247061 5648 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:36:10.249155 5648 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:36:10.257323 5648 catalog_manager.cc:1347] Generated new cluster ID: fab3bd125e4c44ed95219b8bd4e2e991
I20260620 23:36:10.257469 5648 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:36:10.270907 5648 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:36:10.272852 5648 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:36:10.289635 5648 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93162fb61aec41918fc40ebf55fb5a1e: Generated new TSK 0
I20260620 23:36:10.290567 5648 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 23:36:10.338164 5664 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 23:36:10.338357 5664 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 23:36:10.339457 5637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:10.315303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37200 (local address 127.0.97.62:42629)
0620 23:36:10.315506 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:10.315513 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:10.315942 (+ 429us) server_negotiation.cc:407] Connection header received
0620 23:36:10.316272 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:10.316279 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:10.316380 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:10.316547 (+ 167us) server_negotiation.cc:226] Negotiated authn=JWT
0620 23:36:10.317091 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:10.318262 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:10.320049 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:10.320751 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:10.320892 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:10.321481 (+ 589us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 23:36:10.338715 (+ 17234us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45249/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 23:36:10.339133 (+ 418us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37200: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45249/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":82}
W20260620 23:36:10.339581 5637 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37200: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45249/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 23:36:10.341115 5663 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:36:10.315143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42629 (local address 127.0.0.1:37200)
0620 23:36:10.315815 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:10.315831 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:10.315992 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:10.316594 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:10.316597 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:10.316607 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0620 23:36:10.316941 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:10.316950 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:10.318473 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:10.318478 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:10.319895 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:10.319908 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:10.320897 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:10.320959 (+ 62us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:10.321095 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:10.321164 (+ 69us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 23:36:10.339084 (+ 17920us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45249/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 23:36:10.339392 (+ 308us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:42629: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45249/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":365,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.97.62:42629: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45249/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 23:36:10.343250 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 5568
I20260620 23:36:10.344602 5629 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:36:10.445647 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 5568
2026-06-20T23:36:10Z chronyd exiting
I20260620 23:36:10.496292 388 test_util.cc:175] -----------------------------------------------
I20260620 23:36:10.496429 388 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781998525501163-388-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (767 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 23:36:10.500322 388 mini_oidc.cc:152] Starting JWKS server
I20260620 23:36:10.502138 388 webserver.cc:460] Webserver started at https://127.0.0.1:41027/ using document root <none> and password file <none>
I20260620 23:36:10.502345 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 23:36:10.503002 388 webserver.cc:460] Webserver started at http://127.0.0.1:34335/ using document root <none> and password file <none>
2026-06-20T23:36:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:36:10Z Disabled control of system clock
I20260620 23:36:10.532002 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781998525501163-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781998525501163-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781998525501163-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38251
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:46231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38251
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41027/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781998525501163-388-0/testchainca.cert with env {}
W20260620 23:36:10.861893 5671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:36:10.862265 5671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:36:10.862421 5671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:36:10.869747 5671 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:36:10.869938 5671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:36:10.870028 5671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:36:10.870141 5671 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:36:10.870216 5671 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 23:36:10.870316 5671 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 23:36:10.870388 5671 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41027/jwks.json
I20260620 23:36:10.880359 5671 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781998525501163-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781998525501163-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38251
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41027/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781998525501163-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781998525501163-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:36:10.881753 5671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:36:10.883037 5671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:36:10.894958 5676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:10.894994 5677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:10.894910 5679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:36:10.895849 5671 server_base.cc:1034] running on GCE node
I20260620 23:36:10.896704 5671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:36:10.898144 5671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:36:10.899362 5671 hybrid_clock.cc:648] HybridClock initialized: now 1781998570899320 us; error 37 us; skew 500 ppm
I20260620 23:36:10.899729 5671 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:36:10.905368 5671 webserver.cc:460] Webserver started at http://127.0.97.62:40673/ using document root <none> and password file <none>
I20260620 23:36:10.906104 5671 fs_manager.cc:359] Metadata directory not provided
I20260620 23:36:10.906203 5671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:36:10.906526 5671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:36:10.909519 5671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781998525501163-388-0/minicluster-data/master-0/data/instance:
uuid: "e497a37eca9a476d9c96713b27e2a794"
format_stamp: "Formatted at 2026-06-20 23:36:10 on dist-test-slave-tcgx"
server_key: "f29cf61d4a70beb6d10ced149c7ed129"
server_key_iv: "d950340c8bf3a5e318f0b655217ba1a4"
server_key_version: "encryptionkey@0"
I20260620 23:36:10.910311 5671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781998525501163-388-0/minicluster-data/master-0/wal/instance:
uuid: "e497a37eca9a476d9c96713b27e2a794"
format_stamp: "Formatted at 2026-06-20 23:36:10 on dist-test-slave-tcgx"
server_key: "f29cf61d4a70beb6d10ced149c7ed129"
server_key_iv: "d950340c8bf3a5e318f0b655217ba1a4"
server_key_version: "encryptionkey@0"
I20260620 23:36:10.915796 5671 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 23:36:10.919521 5685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:10.921130 5671 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 23:36:10.921363 5671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781998525501163-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781998525501163-388-0/minicluster-data/master-0/wal
uuid: "e497a37eca9a476d9c96713b27e2a794"
format_stamp: "Formatted at 2026-06-20 23:36:10 on dist-test-slave-tcgx"
server_key: "f29cf61d4a70beb6d10ced149c7ed129"
server_key_iv: "d950340c8bf3a5e318f0b655217ba1a4"
server_key_version: "encryptionkey@0"
I20260620 23:36:10.921591 5671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781998525501163-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781998525501163-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781998525501163-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:36:10.935508 5671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:36:10.936415 5671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:36:10.936712 5671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:36:10.956008 5671 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38251
I20260620 23:36:10.956029 5736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38251 every 8 connection(s)
I20260620 23:36:10.957432 5671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:36:10.961580 5737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:36:10.962250 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 5671
I20260620 23:36:10.962610 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781998525501163-388-0/minicluster-data/master-0/wal/instance
I20260620 23:36:10.963310 388 external_mini_cluster.cc:1428] Setting key d8b6dc37605a949cfb26c73eb654fb03
I20260620 23:36:10.975230 5737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e497a37eca9a476d9c96713b27e2a794: Bootstrap starting.
I20260620 23:36:10.977254 5740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:10.965390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50558 (local address 127.0.97.62:38251)
0620 23:36:10.966503 (+ 1113us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:10.966516 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:10.966562 (+ 46us) server_negotiation.cc:407] Connection header received
0620 23:36:10.968272 (+ 1710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:10.968296 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:10.968814 (+ 518us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:10.969254 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:10.969826 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:10.971178 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:10.972545 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:10.973293 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:10.973513 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:10.974748 (+ 1235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:10.974757 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:10.974773 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:36:10.975497 (+ 724us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:10.975580 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:36:10.975602 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:10.975738 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:10.975963 (+ 225us) server_negotiation.cc:299] Negotiation successful
0620 23:36:10.976282 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":409,"threads_started":1}
I20260620 23:36:10.978876 5737 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e497a37eca9a476d9c96713b27e2a794: Neither blocks nor log segments found. Creating new log.
I20260620 23:36:10.980206 5737 log.cc:824] T 00000000000000000000000000000000 P e497a37eca9a476d9c96713b27e2a794: Log is configured to *not* fsync() on all Append() calls
I20260620 23:36:10.981353 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:36:10.983493 5737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e497a37eca9a476d9c96713b27e2a794: No bootstrap required, opened a new log
I20260620 23:36:10.991533 5737 raft_consensus.cc:348] T 00000000000000000000000000000000 P e497a37eca9a476d9c96713b27e2a794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e497a37eca9a476d9c96713b27e2a794" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38251 } }
I20260620 23:36:10.992022 5737 raft_consensus.cc:374] T 00000000000000000000000000000000 P e497a37eca9a476d9c96713b27e2a794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:36:10.992158 5737 raft_consensus.cc:729] T 00000000000000000000000000000000 P e497a37eca9a476d9c96713b27e2a794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e497a37eca9a476d9c96713b27e2a794, State: Initialized, Role: FOLLOWER
I20260620 23:36:10.992875 5737 consensus_queue.cc:260] T 00000000000000000000000000000000 P e497a37eca9a476d9c96713b27e2a794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e497a37eca9a476d9c96713b27e2a794" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38251 } }
I20260620 23:36:10.993152 5737 raft_consensus.cc:388] T 00000000000000000000000000000000 P e497a37eca9a476d9c96713b27e2a794 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:36:10.993319 5737 raft_consensus.cc:482] T 00000000000000000000000000000000 P e497a37eca9a476d9c96713b27e2a794 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:36:10.993532 5737 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e497a37eca9a476d9c96713b27e2a794 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:36:10.996050 5737 raft_consensus.cc:504] T 00000000000000000000000000000000 P e497a37eca9a476d9c96713b27e2a794 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e497a37eca9a476d9c96713b27e2a794" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38251 } }
I20260620 23:36:10.996693 5737 leader_election.cc:302] T 00000000000000000000000000000000 P e497a37eca9a476d9c96713b27e2a794 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e497a37eca9a476d9c96713b27e2a794; no voters:
I20260620 23:36:10.997334 5737 leader_election.cc:288] T 00000000000000000000000000000000 P e497a37eca9a476d9c96713b27e2a794 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:36:10.997479 5742 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e497a37eca9a476d9c96713b27e2a794 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:36:10.998225 5742 raft_consensus.cc:686] T 00000000000000000000000000000000 P e497a37eca9a476d9c96713b27e2a794 [term 1 LEADER]: Becoming Leader. State: Replica: e497a37eca9a476d9c96713b27e2a794, State: Running, Role: LEADER
I20260620 23:36:10.998786 5742 consensus_queue.cc:237] T 00000000000000000000000000000000 P e497a37eca9a476d9c96713b27e2a794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e497a37eca9a476d9c96713b27e2a794" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38251 } }
I20260620 23:36:10.999795 5737 sys_catalog.cc:564] T 00000000000000000000000000000000 P e497a37eca9a476d9c96713b27e2a794 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:36:11.002286 5744 sys_catalog.cc:455] T 00000000000000000000000000000000 P e497a37eca9a476d9c96713b27e2a794 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e497a37eca9a476d9c96713b27e2a794. Latest consensus state: current_term: 1 leader_uuid: "e497a37eca9a476d9c96713b27e2a794" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e497a37eca9a476d9c96713b27e2a794" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38251 } } }
I20260620 23:36:11.002697 5743 sys_catalog.cc:455] T 00000000000000000000000000000000 P e497a37eca9a476d9c96713b27e2a794 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e497a37eca9a476d9c96713b27e2a794" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e497a37eca9a476d9c96713b27e2a794" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38251 } } }
I20260620 23:36:11.002868 5744 sys_catalog.cc:458] T 00000000000000000000000000000000 P e497a37eca9a476d9c96713b27e2a794 [sys.catalog]: This master's current role is: LEADER
I20260620 23:36:11.003000 5743 sys_catalog.cc:458] T 00000000000000000000000000000000 P e497a37eca9a476d9c96713b27e2a794 [sys.catalog]: This master's current role is: LEADER
I20260620 23:36:11.004602 5748 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:36:11.010488 5748 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:36:11.018986 5748 catalog_manager.cc:1347] Generated new cluster ID: 215bebd1825349dc8ed82daeb148af36
I20260620 23:36:11.019121 5748 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:36:11.040297 5748 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:36:11.042413 5748 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:36:11.054446 5748 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e497a37eca9a476d9c96713b27e2a794: Generated new TSK 0
I20260620 23:36:11.055426 5748 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 23:36:11.096050 5767 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 23:36:11.096200 5767 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 23:36:11.097432 5740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:11.076452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50574 (local address 127.0.97.62:38251)
0620 23:36:11.076679 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:11.076686 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:11.077048 (+ 362us) server_negotiation.cc:407] Connection header received
0620 23:36:11.077262 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:11.077268 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:11.077372 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:11.077552 (+ 180us) server_negotiation.cc:226] Negotiated authn=JWT
0620 23:36:11.077992 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:11.079149 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:11.080675 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:11.081363 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:11.081490 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:11.082007 (+ 517us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 23:36:11.096715 (+ 14708us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41027/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 23:36:11.097122 (+ 407us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50574: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41027/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":79}
W20260620 23:36:11.097560 5740 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50574: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41027/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260620 23:36:11.097647 5766 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:36:11.076281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38251 (local address 127.0.0.1:50574)
0620 23:36:11.076943 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:11.076959 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:11.077082 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:11.077569 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:11.077573 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:11.077582 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0620 23:36:11.077831 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:11.077839 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:11.079294 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:11.079298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:11.080513 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:11.080528 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:11.081487 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:11.081530 (+ 43us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:11.081680 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:11.081714 (+ 34us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 23:36:11.097001 (+ 15287us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41027/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 23:36:11.097221 (+ 220us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:38251: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41027/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":372,"threads_started":1}
I20260620 23:36:11.099157 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 5671
I20260620 23:36:11.100441 5732 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:36:11.194152 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 5671
2026-06-20T23:36:11Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (749 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 23:36:11.249435 388 mini_oidc.cc:152] Starting JWKS server
I20260620 23:36:11.251338 388 webserver.cc:460] Webserver started at https://127.0.0.1:33183/ using document root <none> and password file <none>
I20260620 23:36:11.251536 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 23:36:11.252117 388 webserver.cc:460] Webserver started at http://127.0.0.1:36347/ using document root <none> and password file <none>
2026-06-20T23:36:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:36:11Z Disabled control of system clock
I20260620 23:36:11.283264 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781998525501163-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781998525501163-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781998525501163-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37941
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:41147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37941
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33183/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 23:36:11.613287 5774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:36:11.613605 5774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:36:11.613756 5774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:36:11.621218 5774 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:36:11.621335 5774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:36:11.621435 5774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:36:11.621542 5774 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:36:11.621621 5774 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 23:36:11.621728 5774 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 23:36:11.621773 5774 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33183/jwks.json
I20260620 23:36:11.631034 5774 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781998525501163-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781998525501163-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37941
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33183/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781998525501163-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:36:11.632417 5774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:36:11.634229 5774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:36:11.646041 5780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:11.646064 5782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:11.646104 5779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:36:11.646667 5774 server_base.cc:1034] running on GCE node
I20260620 23:36:11.647576 5774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:36:11.649052 5774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:36:11.650274 5774 hybrid_clock.cc:648] HybridClock initialized: now 1781998571650184 us; error 75 us; skew 500 ppm
I20260620 23:36:11.650674 5774 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:36:11.653110 5774 webserver.cc:460] Webserver started at http://127.0.97.62:40733/ using document root <none> and password file <none>
I20260620 23:36:11.653869 5774 fs_manager.cc:359] Metadata directory not provided
I20260620 23:36:11.653968 5774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:36:11.654325 5774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:36:11.657235 5774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781998525501163-388-0/minicluster-data/master-0/data/instance:
uuid: "f621350d414044ac8f367bf94c5c562a"
format_stamp: "Formatted at 2026-06-20 23:36:11 on dist-test-slave-tcgx"
server_key: "17d9ede329900b6126d507bcd4b8983b"
server_key_iv: "39185974dabe2bda9aa96d11b7453d10"
server_key_version: "encryptionkey@0"
I20260620 23:36:11.658234 5774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781998525501163-388-0/minicluster-data/master-0/wal/instance:
uuid: "f621350d414044ac8f367bf94c5c562a"
format_stamp: "Formatted at 2026-06-20 23:36:11 on dist-test-slave-tcgx"
server_key: "17d9ede329900b6126d507bcd4b8983b"
server_key_iv: "39185974dabe2bda9aa96d11b7453d10"
server_key_version: "encryptionkey@0"
I20260620 23:36:11.664726 5774 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260620 23:36:11.668200 5788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:11.669607 5774 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:36:11.669795 5774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781998525501163-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781998525501163-388-0/minicluster-data/master-0/wal
uuid: "f621350d414044ac8f367bf94c5c562a"
format_stamp: "Formatted at 2026-06-20 23:36:11 on dist-test-slave-tcgx"
server_key: "17d9ede329900b6126d507bcd4b8983b"
server_key_iv: "39185974dabe2bda9aa96d11b7453d10"
server_key_version: "encryptionkey@0"
I20260620 23:36:11.670008 5774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781998525501163-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781998525501163-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781998525501163-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:36:11.685568 5774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:36:11.686419 5774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:36:11.686709 5774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:36:11.705500 5774 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37941
I20260620 23:36:11.705538 5839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37941 every 8 connection(s)
I20260620 23:36:11.706881 5774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:36:11.710783 5840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:36:11.712659 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 5774
I20260620 23:36:11.712853 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781998525501163-388-0/minicluster-data/master-0/wal/instance
I20260620 23:36:11.713667 388 external_mini_cluster.cc:1428] Setting key 3df3c7c903ba214b0cff2d96fe92b211
I20260620 23:36:11.722914 5840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f621350d414044ac8f367bf94c5c562a: Bootstrap starting.
I20260620 23:36:11.726302 5840 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f621350d414044ac8f367bf94c5c562a: Neither blocks nor log segments found. Creating new log.
I20260620 23:36:11.727476 5840 log.cc:824] T 00000000000000000000000000000000 P f621350d414044ac8f367bf94c5c562a: Log is configured to *not* fsync() on all Append() calls
I20260620 23:36:11.727525 5843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:11.715731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52616 (local address 127.0.97.62:37941)
0620 23:36:11.716813 (+ 1082us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:11.716825 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:11.716879 (+ 54us) server_negotiation.cc:407] Connection header received
0620 23:36:11.718696 (+ 1817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:11.718721 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:11.719276 (+ 555us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:11.719665 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:11.720264 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:11.721604 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:11.722905 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:11.723696 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:11.723901 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:11.725115 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:11.725122 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:11.725137 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:36:11.725675 (+ 538us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:11.725767 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:36:11.725788 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:11.725930 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:11.726155 (+ 225us) server_negotiation.cc:299] Negotiation successful
0620 23:36:11.726508 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":382,"threads_started":1}
I20260620 23:36:11.730556 5840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f621350d414044ac8f367bf94c5c562a: No bootstrap required, opened a new log
I20260620 23:36:11.731356 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:36:11.737987 5840 raft_consensus.cc:348] T 00000000000000000000000000000000 P f621350d414044ac8f367bf94c5c562a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f621350d414044ac8f367bf94c5c562a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37941 } }
I20260620 23:36:11.738369 5840 raft_consensus.cc:374] T 00000000000000000000000000000000 P f621350d414044ac8f367bf94c5c562a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:36:11.738485 5840 raft_consensus.cc:729] T 00000000000000000000000000000000 P f621350d414044ac8f367bf94c5c562a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f621350d414044ac8f367bf94c5c562a, State: Initialized, Role: FOLLOWER
I20260620 23:36:11.739174 5840 consensus_queue.cc:260] T 00000000000000000000000000000000 P f621350d414044ac8f367bf94c5c562a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f621350d414044ac8f367bf94c5c562a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37941 } }
I20260620 23:36:11.739444 5840 raft_consensus.cc:388] T 00000000000000000000000000000000 P f621350d414044ac8f367bf94c5c562a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:36:11.739603 5840 raft_consensus.cc:482] T 00000000000000000000000000000000 P f621350d414044ac8f367bf94c5c562a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:36:11.739802 5840 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f621350d414044ac8f367bf94c5c562a [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:36:11.741907 5840 raft_consensus.cc:504] T 00000000000000000000000000000000 P f621350d414044ac8f367bf94c5c562a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f621350d414044ac8f367bf94c5c562a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37941 } }
I20260620 23:36:11.742430 5840 leader_election.cc:302] T 00000000000000000000000000000000 P f621350d414044ac8f367bf94c5c562a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f621350d414044ac8f367bf94c5c562a; no voters:
I20260620 23:36:11.743131 5840 leader_election.cc:288] T 00000000000000000000000000000000 P f621350d414044ac8f367bf94c5c562a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:36:11.743278 5845 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f621350d414044ac8f367bf94c5c562a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:36:11.744612 5845 raft_consensus.cc:686] T 00000000000000000000000000000000 P f621350d414044ac8f367bf94c5c562a [term 1 LEADER]: Becoming Leader. State: Replica: f621350d414044ac8f367bf94c5c562a, State: Running, Role: LEADER
I20260620 23:36:11.745218 5845 consensus_queue.cc:237] T 00000000000000000000000000000000 P f621350d414044ac8f367bf94c5c562a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f621350d414044ac8f367bf94c5c562a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37941 } }
I20260620 23:36:11.745492 5840 sys_catalog.cc:564] T 00000000000000000000000000000000 P f621350d414044ac8f367bf94c5c562a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:36:11.748205 5847 sys_catalog.cc:455] T 00000000000000000000000000000000 P f621350d414044ac8f367bf94c5c562a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f621350d414044ac8f367bf94c5c562a. Latest consensus state: current_term: 1 leader_uuid: "f621350d414044ac8f367bf94c5c562a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f621350d414044ac8f367bf94c5c562a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37941 } } }
I20260620 23:36:11.748328 5846 sys_catalog.cc:455] T 00000000000000000000000000000000 P f621350d414044ac8f367bf94c5c562a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f621350d414044ac8f367bf94c5c562a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f621350d414044ac8f367bf94c5c562a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37941 } } }
I20260620 23:36:11.748627 5847 sys_catalog.cc:458] T 00000000000000000000000000000000 P f621350d414044ac8f367bf94c5c562a [sys.catalog]: This master's current role is: LEADER
I20260620 23:36:11.748669 5846 sys_catalog.cc:458] T 00000000000000000000000000000000 P f621350d414044ac8f367bf94c5c562a [sys.catalog]: This master's current role is: LEADER
I20260620 23:36:11.749574 5854 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:36:11.756733 5854 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:36:11.766582 5854 catalog_manager.cc:1347] Generated new cluster ID: ae2a3fc17a3043ea92fd4c28a257eb64
I20260620 23:36:11.766717 5854 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:36:11.780303 5854 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:36:11.781868 5854 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:36:11.790405 5854 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f621350d414044ac8f367bf94c5c562a: Generated new TSK 0
I20260620 23:36:11.791359 5854 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:36:11.820922 5843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:11.812244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52628 (local address 127.0.97.62:37941)
0620 23:36:11.812563 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:11.812571 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:11.812876 (+ 305us) server_negotiation.cc:407] Connection header received
0620 23:36:11.813147 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:11.813151 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:11.813224 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:11.813413 (+ 189us) server_negotiation.cc:226] Negotiated authn=JWT
0620 23:36:11.813996 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:11.814809 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:11.816939 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:11.817425 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:11.817530 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:11.818188 (+ 658us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 23:36:11.820246 (+ 2058us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33183/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 23:36:11.820615 (+ 369us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52628: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33183/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":79}
W20260620 23:36:11.821065 5843 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52628: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33183/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 23:36:11.821166 5869 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:36:11.812110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37941 (local address 127.0.0.1:52628)
0620 23:36:11.812756 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:11.812777 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:11.812948 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:11.813459 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:11.813464 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:11.813478 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0620 23:36:11.813810 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:11.813823 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:11.814993 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:11.814999 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:11.816769 (+ 1770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:11.816786 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:11.817558 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:11.817635 (+ 77us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:11.817837 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:11.817885 (+ 48us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 23:36:11.820470 (+ 2585us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33183/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 23:36:11.820732 (+ 262us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:37941: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33183/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":344,"threads_started":1}
I20260620 23:36:11.822474 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 5774
I20260620 23:36:11.823746 5835 generic_service.cc:196] Checking for leaks (request via RPC)
E20260620 23:36:11.839032 5870 webserver.cc:533] Webserver: sslize failed:
E20260620 23:36:11.839172 5870 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 23:36:11.917162 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 5774
2026-06-20T23:36:11Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (723 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T23:36:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:36:11Z Disabled control of system clock
I20260620 23:36:12.002548 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781998525501163-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781998525501163-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781998525501163-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39441
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:40889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39441
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781998525501163-388-0/testchainca.cert with env {}
W20260620 23:36:12.334805 5875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:36:12.335151 5875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:36:12.335330 5875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:36:12.342819 5875 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:36:12.342991 5875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:36:12.343073 5875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:36:12.343182 5875 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:36:12.343233 5875 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 23:36:12.343339 5875 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 23:36:12.343411 5875 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 23:36:12.353564 5875 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781998525501163-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781998525501163-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39441
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781998525501163-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781998525501163-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:36:12.354974 5875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:36:12.356312 5875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:36:12.368574 5880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:12.368520 5881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:12.368551 5883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:36:12.369256 5875 server_base.cc:1034] running on GCE node
I20260620 23:36:12.370097 5875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:36:12.371996 5875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:36:12.373234 5875 hybrid_clock.cc:648] HybridClock initialized: now 1781998572373188 us; error 36 us; skew 500 ppm
I20260620 23:36:12.373667 5875 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:36:12.376233 5875 webserver.cc:460] Webserver started at http://127.0.97.62:33965/ using document root <none> and password file <none>
I20260620 23:36:12.377004 5875 fs_manager.cc:359] Metadata directory not provided
I20260620 23:36:12.377130 5875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:36:12.377475 5875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:36:12.380573 5875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781998525501163-388-0/minicluster-data/master-0/data/instance:
uuid: "aa22e359b69d49778cf00a1c7cbb894b"
format_stamp: "Formatted at 2026-06-20 23:36:12 on dist-test-slave-tcgx"
server_key: "8e0824b30dc28eb39c0aef1eb0c36548"
server_key_iv: "f956a4298695ac5a5499a7b8a203827d"
server_key_version: "encryptionkey@0"
I20260620 23:36:12.381443 5875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781998525501163-388-0/minicluster-data/master-0/wal/instance:
uuid: "aa22e359b69d49778cf00a1c7cbb894b"
format_stamp: "Formatted at 2026-06-20 23:36:12 on dist-test-slave-tcgx"
server_key: "8e0824b30dc28eb39c0aef1eb0c36548"
server_key_iv: "f956a4298695ac5a5499a7b8a203827d"
server_key_version: "encryptionkey@0"
I20260620 23:36:12.387146 5875 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 23:36:12.390817 5889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:12.392387 5875 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:36:12.392608 5875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781998525501163-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781998525501163-388-0/minicluster-data/master-0/wal
uuid: "aa22e359b69d49778cf00a1c7cbb894b"
format_stamp: "Formatted at 2026-06-20 23:36:12 on dist-test-slave-tcgx"
server_key: "8e0824b30dc28eb39c0aef1eb0c36548"
server_key_iv: "f956a4298695ac5a5499a7b8a203827d"
server_key_version: "encryptionkey@0"
I20260620 23:36:12.392802 5875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781998525501163-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781998525501163-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781998525501163-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:36:12.431732 5875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:36:12.432708 5875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:36:12.432972 5875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:36:12.451985 5875 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39441
I20260620 23:36:12.452006 5940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39441 every 8 connection(s)
I20260620 23:36:12.453434 5875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:36:12.454056 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 5875
I20260620 23:36:12.454309 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781998525501163-388-0/minicluster-data/master-0/wal/instance
I20260620 23:36:12.455176 388 external_mini_cluster.cc:1428] Setting key a4220e9927e8a499b620c5349ae94f62
I20260620 23:36:12.458254 5941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:36:12.470996 5944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:12.457151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38584 (local address 127.0.97.62:39441)
0620 23:36:12.458254 (+ 1103us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:12.458271 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:12.458327 (+ 56us) server_negotiation.cc:407] Connection header received
0620 23:36:12.460627 (+ 2300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:12.460657 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:12.461265 (+ 608us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:12.461759 (+ 494us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:12.462304 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:12.464429 (+ 2125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:12.465722 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:12.466732 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:12.466908 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:12.468108 (+ 1200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:12.468123 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:12.468142 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:36:12.468865 (+ 723us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:12.468973 (+ 108us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:36:12.469004 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:12.469129 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:12.469320 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 23:36:12.469657 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":347,"threads_started":1}
I20260620 23:36:12.472376 5941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa22e359b69d49778cf00a1c7cbb894b: Bootstrap starting.
I20260620 23:36:12.474817 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:36:12.475628 5941 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa22e359b69d49778cf00a1c7cbb894b: Neither blocks nor log segments found. Creating new log.
I20260620 23:36:12.476600 5941 log.cc:824] T 00000000000000000000000000000000 P aa22e359b69d49778cf00a1c7cbb894b: Log is configured to *not* fsync() on all Append() calls
I20260620 23:36:12.479771 5941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa22e359b69d49778cf00a1c7cbb894b: No bootstrap required, opened a new log
I20260620 23:36:12.487861 5941 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa22e359b69d49778cf00a1c7cbb894b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa22e359b69d49778cf00a1c7cbb894b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39441 } }
I20260620 23:36:12.488279 5941 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa22e359b69d49778cf00a1c7cbb894b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:36:12.488382 5941 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa22e359b69d49778cf00a1c7cbb894b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa22e359b69d49778cf00a1c7cbb894b, State: Initialized, Role: FOLLOWER
I20260620 23:36:12.489110 5941 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa22e359b69d49778cf00a1c7cbb894b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa22e359b69d49778cf00a1c7cbb894b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39441 } }
I20260620 23:36:12.489408 5941 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa22e359b69d49778cf00a1c7cbb894b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:36:12.489589 5941 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa22e359b69d49778cf00a1c7cbb894b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:36:12.489776 5941 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa22e359b69d49778cf00a1c7cbb894b [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:36:12.492194 5941 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa22e359b69d49778cf00a1c7cbb894b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa22e359b69d49778cf00a1c7cbb894b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39441 } }
I20260620 23:36:12.492753 5941 leader_election.cc:302] T 00000000000000000000000000000000 P aa22e359b69d49778cf00a1c7cbb894b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa22e359b69d49778cf00a1c7cbb894b; no voters:
I20260620 23:36:12.493368 5941 leader_election.cc:288] T 00000000000000000000000000000000 P aa22e359b69d49778cf00a1c7cbb894b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:36:12.493575 5946 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa22e359b69d49778cf00a1c7cbb894b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:36:12.494261 5946 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa22e359b69d49778cf00a1c7cbb894b [term 1 LEADER]: Becoming Leader. State: Replica: aa22e359b69d49778cf00a1c7cbb894b, State: Running, Role: LEADER
I20260620 23:36:12.494814 5946 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa22e359b69d49778cf00a1c7cbb894b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa22e359b69d49778cf00a1c7cbb894b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39441 } }
I20260620 23:36:12.495776 5941 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa22e359b69d49778cf00a1c7cbb894b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:36:12.498399 5948 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa22e359b69d49778cf00a1c7cbb894b [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa22e359b69d49778cf00a1c7cbb894b. Latest consensus state: current_term: 1 leader_uuid: "aa22e359b69d49778cf00a1c7cbb894b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa22e359b69d49778cf00a1c7cbb894b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39441 } } }
I20260620 23:36:12.498376 5947 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa22e359b69d49778cf00a1c7cbb894b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa22e359b69d49778cf00a1c7cbb894b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa22e359b69d49778cf00a1c7cbb894b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39441 } } }
I20260620 23:36:12.498963 5948 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa22e359b69d49778cf00a1c7cbb894b [sys.catalog]: This master's current role is: LEADER
I20260620 23:36:12.498975 5947 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa22e359b69d49778cf00a1c7cbb894b [sys.catalog]: This master's current role is: LEADER
I20260620 23:36:12.501448 5952 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:36:12.509733 5952 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:36:12.519686 5952 catalog_manager.cc:1347] Generated new cluster ID: 43acd738031d49dda94c9415e3869e40
I20260620 23:36:12.519829 5952 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:36:12.577497 5952 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:36:12.578855 5952 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:36:12.586484 5952 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa22e359b69d49778cf00a1c7cbb894b: Generated new TSK 0
I20260620 23:36:12.587518 5952 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:36:13.034355 5944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:12.606721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38598 (local address 127.0.97.62:39441)
0620 23:36:12.607002 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:12.607011 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:12.607426 (+ 415us) server_negotiation.cc:407] Connection header received
0620 23:36:12.607736 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:12.607741 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:12.607823 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:12.607985 (+ 162us) server_negotiation.cc:226] Negotiated authn=JWT
0620 23:36:12.608537 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:12.609252 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:12.611371 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:12.611881 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:12.611993 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:12.612667 (+ 674us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 23:36:13.033775 (+421108us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 23:36:13.034130 (+ 355us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38598: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":72}
W20260620 23:36:13.034515 5944 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38598: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 23:36:13.034727 5970 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:36:12.606653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39441 (local address 127.0.0.1:38598)
0620 23:36:12.607298 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:12.607318 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:12.607484 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:12.608037 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:12.608042 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:12.608056 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0620 23:36:12.608383 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:12.608394 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:12.609425 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:12.609431 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:12.611220 (+ 1789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:12.611237 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:12.612022 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:12.612083 (+ 61us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:12.612279 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:12.612379 (+ 100us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 23:36:13.034108 (+421729us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 23:36:13.034335 (+ 227us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:39441: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":341,"threads_started":1}
I20260620 23:36:13.036080 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 5875
I20260620 23:36:13.037117 5936 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:36:13.139720 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 5875
2026-06-20T23:36:13Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1203 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (435 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (428 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:36:14 dist-test-slave-tcgx krb5kdc[5985](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:36:14 dist-test-slave-tcgx krb5kdc[5985](info): set up 2 sockets
Jun 20 23:36:14 dist-test-slave-tcgx krb5kdc[5985](info): commencing operation
krb5kdc: starting...
W20260620 23:36:15.095687 388 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.002s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/krb5kdc/test-user.keytab.
Jun 20 23:36:15 dist-test-slave-tcgx krb5kdc[5985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:36:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:36:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 23:36:15.315315 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46221
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46221
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:36:15.647993 5998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:36:15.648308 5998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:36:15.648479 5998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:36:15.656620 5998 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:36:15.656739 5998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:36:15.656848 5998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:36:15.656934 5998 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:36:15.656996 5998 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:36:15.667495 5998 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46221
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:36:15.668926 5998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:36:15.670300 5998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:36:15.683677 6006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:15.683717 6004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:15.683634 6003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:36:15.684640 5998 server_base.cc:1034] running on GCE node
I20260620 23:36:15.685454 5998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:36:15.686890 5998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:36:15.688181 5998 hybrid_clock.cc:648] HybridClock initialized: now 1781998575688127 us; error 39 us; skew 500 ppm
I20260620 23:36:15.688714 5998 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:36:15 dist-test-slave-tcgx krb5kdc[5985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998575, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:36:15.692127 5998 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 23:36:15.693926 5998 webserver.cc:460] Webserver started at http://127.0.97.62:41329/ using document root <none> and password file <none>
I20260620 23:36:15.694716 5998 fs_manager.cc:359] Metadata directory not provided
I20260620 23:36:15.694864 5998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:36:15.695248 5998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:36:15.698244 5998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/minicluster-data/master-0/data/instance:
uuid: "eea6c607da714c008c8dcc86d06029c1"
format_stamp: "Formatted at 2026-06-20 23:36:15 on dist-test-slave-tcgx"
server_key: "e160eea478f51e243e9a5441d1455794"
server_key_iv: "eac8b44d1310b1f72f2e64a718ef5565"
server_key_version: "encryptionkey@0"
I20260620 23:36:15.699120 5998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/minicluster-data/master-0/wal/instance:
uuid: "eea6c607da714c008c8dcc86d06029c1"
format_stamp: "Formatted at 2026-06-20 23:36:15 on dist-test-slave-tcgx"
server_key: "e160eea478f51e243e9a5441d1455794"
server_key_iv: "eac8b44d1310b1f72f2e64a718ef5565"
server_key_version: "encryptionkey@0"
I20260620 23:36:15.704497 5998 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 23:36:15.708115 6013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:15.709486 5998 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 23:36:15.709684 5998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/minicluster-data/master-0/wal
uuid: "eea6c607da714c008c8dcc86d06029c1"
format_stamp: "Formatted at 2026-06-20 23:36:15 on dist-test-slave-tcgx"
server_key: "e160eea478f51e243e9a5441d1455794"
server_key_iv: "eac8b44d1310b1f72f2e64a718ef5565"
server_key_version: "encryptionkey@0"
I20260620 23:36:15.709892 5998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:36:15.744849 5998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:36:15.749441 5998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:36:15.749739 5998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:36:15.768926 5998 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46221
I20260620 23:36:15.768955 6064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46221 every 8 connection(s)
I20260620 23:36:15.770335 5998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:36:15.774561 6065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:36:15.776898 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 5998
I20260620 23:36:15.777135 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/minicluster-data/master-0/wal/instance
I20260620 23:36:15.777879 388 external_mini_cluster.cc:1428] Setting key cb4ac48e52df340e14b07e6bfb6f7dbe
I20260620 23:36:15.787179 6065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eea6c607da714c008c8dcc86d06029c1: Bootstrap starting.
I20260620 23:36:15.790271 6065 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eea6c607da714c008c8dcc86d06029c1: Neither blocks nor log segments found. Creating new log.
I20260620 23:36:15.791352 6065 log.cc:824] T 00000000000000000000000000000000 P eea6c607da714c008c8dcc86d06029c1: Log is configured to *not* fsync() on all Append() calls
Jun 20 23:36:15 dist-test-slave-tcgx krb5kdc[5985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:36:15.794566 6065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eea6c607da714c008c8dcc86d06029c1: No bootstrap required, opened a new log
I20260620 23:36:15.801620 6068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:15.780164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33992 (local address 127.0.97.62:46221)
0620 23:36:15.780990 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:15.781003 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:15.781045 (+ 42us) server_negotiation.cc:407] Connection header received
0620 23:36:15.783376 (+ 2331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:15.783408 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:15.783828 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:15.784276 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:15.785797 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:15.787225 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:15.788547 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:15.789245 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:15.789441 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:15.793919 (+ 4478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:15.793927 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:15.793942 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:15.793976 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:15.797216 (+ 3240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:15.798074 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:15.798080 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:15.798087 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:15.798258 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:15.798689 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:15.798694 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:15.798696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:15.799368 (+ 672us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:15.799618 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:15.800084 (+ 466us) server_negotiation.cc:299] Negotiation successful
0620 23:36:15.800492 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":359,"threads_started":1}
I20260620 23:36:15.803454 6065 raft_consensus.cc:348] T 00000000000000000000000000000000 P eea6c607da714c008c8dcc86d06029c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea6c607da714c008c8dcc86d06029c1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46221 } }
I20260620 23:36:15.803824 6065 raft_consensus.cc:374] T 00000000000000000000000000000000 P eea6c607da714c008c8dcc86d06029c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:36:15.803944 6065 raft_consensus.cc:729] T 00000000000000000000000000000000 P eea6c607da714c008c8dcc86d06029c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eea6c607da714c008c8dcc86d06029c1, State: Initialized, Role: FOLLOWER
I20260620 23:36:15.804529 6065 consensus_queue.cc:260] T 00000000000000000000000000000000 P eea6c607da714c008c8dcc86d06029c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea6c607da714c008c8dcc86d06029c1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46221 } }
I20260620 23:36:15.804770 6065 raft_consensus.cc:388] T 00000000000000000000000000000000 P eea6c607da714c008c8dcc86d06029c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:36:15.804927 6065 raft_consensus.cc:482] T 00000000000000000000000000000000 P eea6c607da714c008c8dcc86d06029c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:36:15.805099 6065 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eea6c607da714c008c8dcc86d06029c1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:36:15.807276 6065 raft_consensus.cc:504] T 00000000000000000000000000000000 P eea6c607da714c008c8dcc86d06029c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea6c607da714c008c8dcc86d06029c1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46221 } }
I20260620 23:36:15.807757 6065 leader_election.cc:302] T 00000000000000000000000000000000 P eea6c607da714c008c8dcc86d06029c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eea6c607da714c008c8dcc86d06029c1; no voters:
I20260620 23:36:15.808334 6065 leader_election.cc:288] T 00000000000000000000000000000000 P eea6c607da714c008c8dcc86d06029c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:36:15.808537 6070 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eea6c607da714c008c8dcc86d06029c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:36:15.809566 6070 raft_consensus.cc:686] T 00000000000000000000000000000000 P eea6c607da714c008c8dcc86d06029c1 [term 1 LEADER]: Becoming Leader. State: Replica: eea6c607da714c008c8dcc86d06029c1, State: Running, Role: LEADER
I20260620 23:36:15.810060 6070 consensus_queue.cc:237] T 00000000000000000000000000000000 P eea6c607da714c008c8dcc86d06029c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea6c607da714c008c8dcc86d06029c1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46221 } }
I20260620 23:36:15.810527 6065 sys_catalog.cc:564] T 00000000000000000000000000000000 P eea6c607da714c008c8dcc86d06029c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:36:15.813169 6072 sys_catalog.cc:455] T 00000000000000000000000000000000 P eea6c607da714c008c8dcc86d06029c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eea6c607da714c008c8dcc86d06029c1. Latest consensus state: current_term: 1 leader_uuid: "eea6c607da714c008c8dcc86d06029c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea6c607da714c008c8dcc86d06029c1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46221 } } }
I20260620 23:36:15.813679 6072 sys_catalog.cc:458] T 00000000000000000000000000000000 P eea6c607da714c008c8dcc86d06029c1 [sys.catalog]: This master's current role is: LEADER
I20260620 23:36:15.813211 6071 sys_catalog.cc:455] T 00000000000000000000000000000000 P eea6c607da714c008c8dcc86d06029c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eea6c607da714c008c8dcc86d06029c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea6c607da714c008c8dcc86d06029c1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46221 } } }
I20260620 23:36:15.815157 6071 sys_catalog.cc:458] T 00000000000000000000000000000000 P eea6c607da714c008c8dcc86d06029c1 [sys.catalog]: This master's current role is: LEADER
I20260620 23:36:15.817515 6075 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:36:15.823042 6075 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:36:15.832020 6075 catalog_manager.cc:1347] Generated new cluster ID: 8b843b3ecbef4444842e7d43dc5fa823
I20260620 23:36:15.832180 6075 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:36:15.842303 6075 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:36:15.843743 6075 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:36:15.853281 6075 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eea6c607da714c008c8dcc86d06029c1: Generated new TSK 0
I20260620 23:36:15.854357 6075 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 23:36:15.909916 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46221
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:36:16.242795 6091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:36:16.243116 6091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:36:16.243294 6091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:36:16.250547 6091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:36:16.250703 6091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:36:16.250890 6091 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 23:36:16.260888 6091 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46221
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:36:16.262292 6091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:36:16.263674 6091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:36:16.279628 6099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:16.279654 6097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:16.279628 6096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:36:16.280957 6091 server_base.cc:1034] running on GCE node
I20260620 23:36:16.281656 6091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:36:16.282917 6091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:36:16.284202 6091 hybrid_clock.cc:648] HybridClock initialized: now 1781998576284157 us; error 55 us; skew 500 ppm
I20260620 23:36:16.284566 6091 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:36:16 dist-test-slave-tcgx krb5kdc[5985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998576, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:36:16.291646 6091 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 23:36:16.293356 6091 webserver.cc:460] Webserver started at http://127.0.97.1:37509/ using document root <none> and password file <none>
I20260620 23:36:16.294085 6091 fs_manager.cc:359] Metadata directory not provided
I20260620 23:36:16.294231 6091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:36:16.294548 6091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:36:16.297559 6091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/minicluster-data/ts-0/data/instance:
uuid: "c19dd851239e4c68a0057364ed329a3c"
format_stamp: "Formatted at 2026-06-20 23:36:16 on dist-test-slave-tcgx"
server_key: "b3607bd04fa08239a1d98ceedd4c17bc"
server_key_iv: "eafa6ed53f0bb86460e13fb7dae59e2f"
server_key_version: "encryptionkey@0"
I20260620 23:36:16.298367 6091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/minicluster-data/ts-0/wal/instance:
uuid: "c19dd851239e4c68a0057364ed329a3c"
format_stamp: "Formatted at 2026-06-20 23:36:16 on dist-test-slave-tcgx"
server_key: "b3607bd04fa08239a1d98ceedd4c17bc"
server_key_iv: "eafa6ed53f0bb86460e13fb7dae59e2f"
server_key_version: "encryptionkey@0"
I20260620 23:36:16.303779 6091 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 23:36:16.307380 6106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:16.308736 6091 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260620 23:36:16.308923 6091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/minicluster-data/ts-0/wal
uuid: "c19dd851239e4c68a0057364ed329a3c"
format_stamp: "Formatted at 2026-06-20 23:36:16 on dist-test-slave-tcgx"
server_key: "b3607bd04fa08239a1d98ceedd4c17bc"
server_key_iv: "eafa6ed53f0bb86460e13fb7dae59e2f"
server_key_version: "encryptionkey@0"
I20260620 23:36:16.309150 6091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:36:16.326519 6091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:36:16.331269 6091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:36:16.331569 6091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:36:16.332602 6091 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:36:16.333967 6091 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:36:16.334071 6091 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:16.334218 6091 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:36:16.334288 6091 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:16.367569 6091 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38999
I20260620 23:36:16.367626 6218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38999 every 8 connection(s)
I20260620 23:36:16.368882 6091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
I20260620 23:36:16.375029 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 6091
I20260620 23:36:16.375250 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/minicluster-data/ts-0/wal/instance
I20260620 23:36:16.376048 388 external_mini_cluster.cc:1428] Setting key 994a51fa658aa8138bf3a6c4f7663d96
Jun 20 23:36:16 dist-test-slave-tcgx krb5kdc[5985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998576, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:36:16.390858 6221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:16.371646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50045 (local address 127.0.97.62:46221)
0620 23:36:16.372077 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:16.372082 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:16.373106 (+ 1024us) server_negotiation.cc:407] Connection header received
0620 23:36:16.374994 (+ 1888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:16.374999 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:16.375095 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:16.375235 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:16.377257 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:16.378070 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:16.379759 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:16.380166 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:16.380305 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:16.384343 (+ 4038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:16.384348 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:16.384352 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:16.384399 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:16.387223 (+ 2824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:16.388255 (+ 1032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:16.388260 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:16.388263 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:16.388448 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:16.388973 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:16.388977 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:16.388979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:16.389243 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:16.389359 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:16.390358 (+ 999us) server_negotiation.cc:299] Negotiation successful
0620 23:36:16.390584 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":211,"threads_started":1}
I20260620 23:36:16.392603 6222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:16.372177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46221 (local address 127.0.97.1:50045)
0620 23:36:16.372932 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:16.372971 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:16.374677 (+ 1706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:16.375594 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:16.375607 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:16.376393 (+ 786us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:16.377078 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:16.377097 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:16.378247 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:16.378251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:16.379609 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:16.379621 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:16.380400 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:16.380408 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:16.380649 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:16.381474 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:16.381507 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:16.384075 (+ 2568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:36:16.387381 (+ 3306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:16.387394 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:16.387417 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:16.388097 (+ 680us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:16.388575 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:16.388583 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:16.388589 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:16.388851 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:16.389379 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:36:16.389390 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:36:16.389892 (+ 502us) client_negotiation.cc:769] Sending connection context
0620 23:36:16.390252 (+ 360us) client_negotiation.cc:240] Negotiation successful
0620 23:36:16.390863 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":299,"threads_started":1}
I20260620 23:36:16.394372 6219 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46221
I20260620 23:36:16.394743 6219 heartbeater.cc:464] Registering TS with master...
I20260620 23:36:16.395592 6219 heartbeater.cc:511] Master 127.0.97.62:46221 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 23:36:16.397881 6030 ts_manager.cc:194] Registered new tserver with Master: c19dd851239e4c68a0057364ed329a3c (127.0.97.1:38999)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260620 23:36:16.399619 6030 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:50045
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 23:36:16.423669 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46221
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:36:16.755437 6225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:36:16.755718 6225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:36:16.755811 6225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:36:16.763044 6225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:36:16.763163 6225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:36:16.763278 6225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 23:36:16.776228 6225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46221
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:36:16.777848 6225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:36:16.779567 6225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:36:16.795364 6230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:16.795389 6233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:16.796129 6231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:36:16.796348 6225 server_base.cc:1034] running on GCE node
I20260620 23:36:16.797016 6225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:36:16.798040 6225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:36:16.799270 6225 hybrid_clock.cc:648] HybridClock initialized: now 1781998576799205 us; error 62 us; skew 500 ppm
I20260620 23:36:16.799638 6225 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:36:16 dist-test-slave-tcgx krb5kdc[5985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998576, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:36:16.802778 6225 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 23:36:16.804278 6225 webserver.cc:460] Webserver started at http://127.0.97.2:43739/ using document root <none> and password file <none>
I20260620 23:36:16.804975 6225 fs_manager.cc:359] Metadata directory not provided
I20260620 23:36:16.805095 6225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:36:16.805341 6225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:36:16.808444 6225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/minicluster-data/ts-1/data/instance:
uuid: "5783aa32101e4166917d0316d9c6c941"
format_stamp: "Formatted at 2026-06-20 23:36:16 on dist-test-slave-tcgx"
server_key: "4e11362fd95de3b7b8fc3cbdd5ce648b"
server_key_iv: "b465b2099082356cead756ea0ba88e22"
server_key_version: "encryptionkey@0"
I20260620 23:36:16.809158 6225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/minicluster-data/ts-1/wal/instance:
uuid: "5783aa32101e4166917d0316d9c6c941"
format_stamp: "Formatted at 2026-06-20 23:36:16 on dist-test-slave-tcgx"
server_key: "4e11362fd95de3b7b8fc3cbdd5ce648b"
server_key_iv: "b465b2099082356cead756ea0ba88e22"
server_key_version: "encryptionkey@0"
I20260620 23:36:16.814998 6225 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 23:36:16.818795 6240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:16.820586 6225 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 23:36:16.820732 6225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/minicluster-data/ts-1/wal
uuid: "5783aa32101e4166917d0316d9c6c941"
format_stamp: "Formatted at 2026-06-20 23:36:16 on dist-test-slave-tcgx"
server_key: "4e11362fd95de3b7b8fc3cbdd5ce648b"
server_key_iv: "b465b2099082356cead756ea0ba88e22"
server_key_version: "encryptionkey@0"
I20260620 23:36:16.820916 6225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:36:16.837425 6225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:36:16.841993 6225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:36:16.842249 6225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:36:16.843221 6225 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:36:16.844570 6225 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:36:16.844641 6225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:16.844728 6225 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:36:16.844771 6225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:16.876271 6225 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37263
I20260620 23:36:16.876302 6352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37263 every 8 connection(s)
I20260620 23:36:16.877812 6225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
I20260620 23:36:16.884860 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 6225
I20260620 23:36:16.885070 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/minicluster-data/ts-1/wal/instance
I20260620 23:36:16.885519 388 external_mini_cluster.cc:1428] Setting key 643b1c05f377c99d92d61697ffe44ea1
Jun 20 23:36:16 dist-test-slave-tcgx krb5kdc[5985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998576, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:36:16.900050 6221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:16.880494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34853 (local address 127.0.97.62:46221)
0620 23:36:16.880675 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:16.880679 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:16.881977 (+ 1298us) server_negotiation.cc:407] Connection header received
0620 23:36:16.883842 (+ 1865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:16.883846 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:16.883916 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:16.884014 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:16.885853 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:16.886785 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:16.889212 (+ 2427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:16.889684 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:16.889785 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:16.894055 (+ 4270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:16.894063 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:16.894071 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:16.894118 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:16.896709 (+ 2591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:16.897687 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:16.897691 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:16.897693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:16.897812 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:16.898301 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:16.898304 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:16.898306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:16.898554 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:16.898657 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:16.899639 (+ 982us) server_negotiation.cc:299] Negotiation successful
0620 23:36:16.899806 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 23:36:16.901643 6355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:16.881065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46221 (local address 127.0.97.2:34853)
0620 23:36:16.881806 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:16.881844 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:16.883582 (+ 1738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:16.884277 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:16.884286 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:16.884880 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:16.885672 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:16.885692 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:16.887025 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:16.887029 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:16.889046 (+ 2017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:16.889065 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:16.889923 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:16.889934 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:16.890271 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:16.891055 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:16.891090 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:16.893802 (+ 2712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:36:16.896858 (+ 3056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:16.896871 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:16.896893 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:16.897550 (+ 657us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:16.897920 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:16.897925 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:16.897929 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:16.898179 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:16.898675 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:36:16.898686 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:36:16.899207 (+ 521us) client_negotiation.cc:769] Sending connection context
0620 23:36:16.899524 (+ 317us) client_negotiation.cc:240] Negotiation successful
0620 23:36:16.900113 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":311,"threads_started":1}
I20260620 23:36:16.903276 6353 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46221
I20260620 23:36:16.903599 6353 heartbeater.cc:464] Registering TS with master...
I20260620 23:36:16.904341 6353 heartbeater.cc:511] Master 127.0.97.62:46221 requested a full tablet report, sending...
I20260620 23:36:16.905783 6030 ts_manager.cc:194] Registered new tserver with Master: 5783aa32101e4166917d0316d9c6c941 (127.0.97.2:37263)
I20260620 23:36:16.906587 6030 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:34853
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 23:36:16.934542 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46221
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:36:17.265327 6358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:36:17.265643 6358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:36:17.265738 6358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:36:17.273114 6358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:36:17.273239 6358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:36:17.273353 6358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 23:36:17.283306 6358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46221
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:36:17.284644 6358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:36:17.285943 6358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:36:17.299512 6363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:17.299446 6364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:17.299490 6366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:36:17.300331 6358 server_base.cc:1034] running on GCE node
I20260620 23:36:17.301025 6358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:36:17.302013 6358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:36:17.303241 6358 hybrid_clock.cc:648] HybridClock initialized: now 1781998577303191 us; error 46 us; skew 500 ppm
I20260620 23:36:17.303579 6358 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:36:17 dist-test-slave-tcgx krb5kdc[5985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998577, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:36:17.306826 6358 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 23:36:17.308488 6358 webserver.cc:460] Webserver started at http://127.0.97.3:39293/ using document root <none> and password file <none>
I20260620 23:36:17.309221 6358 fs_manager.cc:359] Metadata directory not provided
I20260620 23:36:17.309369 6358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:36:17.309686 6358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:36:17.313031 6358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/minicluster-data/ts-2/data/instance:
uuid: "28085f146d824f548f86c8cf6eb40967"
format_stamp: "Formatted at 2026-06-20 23:36:17 on dist-test-slave-tcgx"
server_key: "9af3a127011f95ce6bc06756cbdbbcd5"
server_key_iv: "259cc22db045cdadc07666ea082373f0"
server_key_version: "encryptionkey@0"
I20260620 23:36:17.313901 6358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/minicluster-data/ts-2/wal/instance:
uuid: "28085f146d824f548f86c8cf6eb40967"
format_stamp: "Formatted at 2026-06-20 23:36:17 on dist-test-slave-tcgx"
server_key: "9af3a127011f95ce6bc06756cbdbbcd5"
server_key_iv: "259cc22db045cdadc07666ea082373f0"
server_key_version: "encryptionkey@0"
I20260620 23:36:17.319720 6358 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 23:36:17.323518 6373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:17.325117 6358 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 23:36:17.325300 6358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/minicluster-data/ts-2/wal
uuid: "28085f146d824f548f86c8cf6eb40967"
format_stamp: "Formatted at 2026-06-20 23:36:17 on dist-test-slave-tcgx"
server_key: "9af3a127011f95ce6bc06756cbdbbcd5"
server_key_iv: "259cc22db045cdadc07666ea082373f0"
server_key_version: "encryptionkey@0"
I20260620 23:36:17.325527 6358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:36:17.361111 6358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:36:17.365777 6358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:36:17.366098 6358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:36:17.367262 6358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:36:17.368769 6358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:36:17.368878 6358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:17.369041 6358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:36:17.369114 6358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:17.402748 6219 heartbeater.cc:503] Master 127.0.97.62:46221 was elected leader, sending a full tablet report...
I20260620 23:36:17.404160 6358 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34729
I20260620 23:36:17.404259 6485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34729 every 8 connection(s)
I20260620 23:36:17.405635 6358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
I20260620 23:36:17.409448 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 6358
I20260620 23:36:17.409646 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/minicluster-data/ts-2/wal/instance
I20260620 23:36:17.410151 388 external_mini_cluster.cc:1428] Setting key b0d98b0d2b35bfe441ea4d7ce1f196ff
Jun 20 23:36:17 dist-test-slave-tcgx krb5kdc[5985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998577, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:36:17.427356 6488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:17.408228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42535 (local address 127.0.97.62:46221)
0620 23:36:17.408742 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:17.408746 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:17.410004 (+ 1258us) server_negotiation.cc:407] Connection header received
0620 23:36:17.412436 (+ 2432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:17.412440 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:17.412518 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:17.412655 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:17.414393 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:17.415369 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:17.416972 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:17.417362 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:17.417473 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:17.421193 (+ 3720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:17.421199 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:17.421202 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:17.421235 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:17.424022 (+ 2787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:17.425031 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:17.425036 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:17.425040 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:17.425204 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:17.425708 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:17.425712 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:17.425715 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:17.426080 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:17.426189 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:17.426864 (+ 675us) server_negotiation.cc:299] Negotiation successful
0620 23:36:17.427098 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":303,"threads_started":1}
I20260620 23:36:17.428417 6489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:17.408888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46221 (local address 127.0.97.3:42535)
0620 23:36:17.409839 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:17.409881 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:17.412149 (+ 2268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:17.412945 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:17.412958 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:17.413614 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:17.414215 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:17.414234 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:17.415509 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:17.415515 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:17.416798 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:17.416810 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:17.417572 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:17.417577 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:17.417875 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:17.418652 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:17.418680 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:17.420903 (+ 2223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:36:17.424212 (+ 3309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:17.424222 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:17.424243 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:17.424879 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:17.425345 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:17.425350 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:17.425353 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:17.425600 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:17.426194 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:36:17.426201 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:36:17.426540 (+ 339us) client_negotiation.cc:769] Sending connection context
0620 23:36:17.426789 (+ 249us) client_negotiation.cc:240] Negotiation successful
0620 23:36:17.427261 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":381,"threads_started":1}
I20260620 23:36:17.429889 6486 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46221
I20260620 23:36:17.430245 6486 heartbeater.cc:464] Registering TS with master...
I20260620 23:36:17.431185 6486 heartbeater.cc:511] Master 127.0.97.62:46221 requested a full tablet report, sending...
I20260620 23:36:17.432780 6030 ts_manager.cc:194] Registered new tserver with Master: 28085f146d824f548f86c8cf6eb40967 (127.0.97.3:34729)
I20260620 23:36:17.433621 6030 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:42535
I20260620 23:36:17.438715 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/krb5kdc/test-admin.keytab.
Jun 20 23:36:17 dist-test-slave-tcgx krb5kdc[5985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:36:17.465036 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 23:36:17.470278 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/krb5kdc/krb5cc' of type FILE
W20260620 23:36:17.470314 6494 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:36:17.467676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41678 (local address 127.0.97.1:38999)
0620 23:36:17.468408 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:17.468417 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:17.468472 (+ 55us) server_negotiation.cc:407] Connection header received
0620 23:36:17.468534 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:17.468540 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:17.468779 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:17.468901 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:17.469795 (+ 894us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41678: BlockingRecv error: recv got EOF from 127.0.0.1:41678 (error 108)
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":438,"threads_started":1}
Jun 20 23:36:17 dist-test-slave-tcgx krb5kdc[5985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:36:17.471943 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 23:36:17 dist-test-slave-tcgx krb5kdc[5985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:36:17.473340 388 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 23:36:17 dist-test-slave-tcgx krb5kdc[5985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 23:36:17.489060 6494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:17.474993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41682 (local address 127.0.97.1:38999)
0620 23:36:17.475206 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:17.475212 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:17.475418 (+ 206us) server_negotiation.cc:407] Connection header received
0620 23:36:17.475554 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:17.475558 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:17.475619 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:17.475701 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:17.476713 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:17.477687 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:17.478862 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:17.479515 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:17.479611 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:17.483207 (+ 3596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:17.483219 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:17.483234 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:17.483275 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:17.486277 (+ 3002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:17.487105 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:17.487113 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:17.487116 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:17.487234 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:17.487627 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:17.487630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:17.487632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:17.487985 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:17.488202 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:17.488503 (+ 301us) server_negotiation.cc:299] Negotiation successful
0620 23:36:17.488772 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260620 23:36:17.495353 6494 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:36:17.493523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41688 (local address 127.0.97.1:38999)
0620 23:36:17.493700 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:17.493706 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:17.493959 (+ 253us) server_negotiation.cc:407] Connection header received
0620 23:36:17.494097 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:17.494101 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:17.494170 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:17.494253 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:17.495073 (+ 820us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41688: BlockingRecv error: recv got EOF from 127.0.0.1:41688 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260620 23:36:17.495498 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/krb5kdc/krb5cc' of type FILE
Jun 20 23:36:17 dist-test-slave-tcgx krb5kdc[5985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:36:17.497159 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 23:36:17 dist-test-slave-tcgx krb5kdc[5985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:36:17.498648 388 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 23:36:17 dist-test-slave-tcgx krb5kdc[5985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 23:36:17.515641 6494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:17.500277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41692 (local address 127.0.97.1:38999)
0620 23:36:17.500540 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:17.500545 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:17.500639 (+ 94us) server_negotiation.cc:407] Connection header received
0620 23:36:17.500821 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:17.500829 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:17.500892 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:17.500992 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:17.501894 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:17.502625 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:17.503902 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:17.504309 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:17.504421 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:17.508713 (+ 4292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:17.508719 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:17.508723 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:17.508759 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:17.512509 (+ 3750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:17.513586 (+ 1077us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:17.513594 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:17.513599 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:17.513784 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:17.514174 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:17.514179 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:17.514182 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:17.514573 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:17.514783 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:17.515091 (+ 308us) server_negotiation.cc:299] Negotiation successful
0620 23:36:17.515257 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
W20260620 23:36:17.520658 6494 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:36:17.518777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41698 (local address 127.0.97.1:38999)
0620 23:36:17.519051 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:17.519059 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:17.519304 (+ 245us) server_negotiation.cc:407] Connection header received
0620 23:36:17.519494 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:17.519499 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:17.519579 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:17.519682 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:17.520389 (+ 707us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41698: BlockingRecv error: recv got EOF from 127.0.0.1:41698 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260620 23:36:17.521266 388 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/krb5kdc/krb5cc' of type FILE: Runtime error: Bad format in credentials cache (filename: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998525501163-388-0/krb5kdc/krb5cc)
Jun 20 23:36:17 dist-test-slave-tcgx krb5kdc[5985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:36:17.522859 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 23:36:17 dist-test-slave-tcgx krb5kdc[5985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:36:17.524291 388 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 23:36:17 dist-test-slave-tcgx krb5kdc[5985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 23:36:17.539160 6494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:17.525746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41702 (local address 127.0.97.1:38999)
0620 23:36:17.525972 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:17.525980 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:17.526183 (+ 203us) server_negotiation.cc:407] Connection header received
0620 23:36:17.526387 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:17.526391 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:17.526449 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:17.526549 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:17.527594 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:17.528303 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:17.529726 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:17.530138 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:17.530286 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:17.533723 (+ 3437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:17.533727 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:17.533731 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:17.533763 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:17.536569 (+ 2806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:17.537253 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:17.537258 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:17.537260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:17.537408 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:17.537772 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:17.537776 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:17.537778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:17.538055 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:17.538282 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:17.538607 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 23:36:17.538777 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 23:36:17.540997 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 6091
I20260620 23:36:17.553112 6494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:17.541788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41710 (local address 127.0.97.1:38999)
0620 23:36:17.542016 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:17.542023 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:17.542246 (+ 223us) server_negotiation.cc:407] Connection header received
0620 23:36:17.542351 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:17.542354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:17.542409 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:17.542497 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:17.543434 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:17.544105 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:17.545414 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:17.545860 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:17.545998 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:17.548081 (+ 2083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:17.548089 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:17.548093 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:17.548131 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:17.550466 (+ 2335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:17.551187 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:17.551193 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:17.551196 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:17.551339 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:17.551716 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:17.551721 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:17.551724 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:17.551971 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:17.552260 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:17.552581 (+ 321us) server_negotiation.cc:299] Negotiation successful
0620 23:36:17.552799 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 23:36:17.553685 6214 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:36:17.637696 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 6091
I20260620 23:36:17.654747 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 6225
Jun 20 23:36:17 dist-test-slave-tcgx krb5kdc[5985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 23:36:17.670538 6507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:17.655737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57406 (local address 127.0.97.2:37263)
0620 23:36:17.656289 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:17.656304 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:17.656331 (+ 27us) server_negotiation.cc:407] Connection header received
0620 23:36:17.656393 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:17.656400 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:17.656613 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:17.656774 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:17.657773 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:17.658768 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:17.659983 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:17.660517 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:17.660614 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:17.664373 (+ 3759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:17.664385 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:17.664458 (+ 73us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:17.664495 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:17.667821 (+ 3326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:17.668539 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:17.668546 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:17.668548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:17.668687 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:17.669126 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:17.669131 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:17.669133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:17.669525 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:17.669765 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:17.670029 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 23:36:17.670220 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":298,"threads_started":1}
I20260620 23:36:17.671890 6348 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:36:17.756677 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 6225
I20260620 23:36:17.772743 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 6358
Jun 20 23:36:17 dist-test-slave-tcgx krb5kdc[5985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 23:36:17.788889 6509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:17.773959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53418 (local address 127.0.97.3:34729)
0620 23:36:17.774612 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:17.774619 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:17.774648 (+ 29us) server_negotiation.cc:407] Connection header received
0620 23:36:17.774707 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:17.774713 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:17.774920 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:17.775184 (+ 264us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:17.776256 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:17.777283 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:17.778562 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:17.779245 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:17.779382 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:17.783264 (+ 3882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:17.783277 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:17.783291 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:17.783333 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:17.786066 (+ 2733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:17.786790 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:17.786797 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:17.786800 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:17.786918 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:17.787339 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:17.787343 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:17.787344 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:17.787726 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:17.787959 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:17.788279 (+ 320us) server_negotiation.cc:299] Negotiation successful
0620 23:36:17.788592 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":344,"threads_started":1}
I20260620 23:36:17.790203 6481 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:36:17.874092 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 6358
I20260620 23:36:17.890241 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 5998
I20260620 23:36:17.891625 6060 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:36:17.994297 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 5998
2026-06-20T23:36:18Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3997 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:36:18 dist-test-slave-tcgx krb5kdc[6514](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:36:18 dist-test-slave-tcgx krb5kdc[6514](info): set up 2 sockets
Jun 20 23:36:18 dist-test-slave-tcgx krb5kdc[6514](info): commencing operation
krb5kdc: starting...
W20260620 23:36:19.095901 388 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/krb5kdc/test-user.keytab.
Jun 20 23:36:19 dist-test-slave-tcgx krb5kdc[6514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:36:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:36:19Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260620 23:36:19.318089 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33681
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33681
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:36:19.649734 6527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:36:19.650102 6527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:36:19.650271 6527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:36:19.657735 6527 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:36:19.657897 6527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:36:19.658016 6527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:36:19.658114 6527 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:36:19.658200 6527 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:36:19.668262 6527 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33681
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:36:19.669721 6527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:36:19.671132 6527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:36:19.683228 6532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:19.683233 6533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:19.683207 6535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:36:19.683691 6527 server_base.cc:1034] running on GCE node
I20260620 23:36:19.684716 6527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:36:19.686215 6527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:36:19.687444 6527 hybrid_clock.cc:648] HybridClock initialized: now 1781998579687393 us; error 46 us; skew 500 ppm
I20260620 23:36:19.687796 6527 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:36:19 dist-test-slave-tcgx krb5kdc[6514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998579, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:36:19.691473 6527 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260620 23:36:19.693408 6527 webserver.cc:460] Webserver started at http://127.0.97.62:38835/ using document root <none> and password file <none>
I20260620 23:36:19.694178 6527 fs_manager.cc:359] Metadata directory not provided
I20260620 23:36:19.694324 6527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:36:19.694650 6527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:36:19.697850 6527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/master-0/data/instance:
uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf"
format_stamp: "Formatted at 2026-06-20 23:36:19 on dist-test-slave-tcgx"
server_key: "c78b515528bab2a281bb762cb781c015"
server_key_iv: "50872f74da3002d8fb660eb1dd36505d"
server_key_version: "encryptionkey@0"
I20260620 23:36:19.698690 6527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/master-0/wal/instance:
uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf"
format_stamp: "Formatted at 2026-06-20 23:36:19 on dist-test-slave-tcgx"
server_key: "c78b515528bab2a281bb762cb781c015"
server_key_iv: "50872f74da3002d8fb660eb1dd36505d"
server_key_version: "encryptionkey@0"
I20260620 23:36:19.704373 6527 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 23:36:19.708122 6542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:19.709498 6527 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:36:19.709699 6527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/master-0/wal
uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf"
format_stamp: "Formatted at 2026-06-20 23:36:19 on dist-test-slave-tcgx"
server_key: "c78b515528bab2a281bb762cb781c015"
server_key_iv: "50872f74da3002d8fb660eb1dd36505d"
server_key_version: "encryptionkey@0"
I20260620 23:36:19.709905 6527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:36:19.733436 6527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:36:19.737633 6527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:36:19.737922 6527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:36:19.757086 6527 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33681
I20260620 23:36:19.757120 6593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33681 every 8 connection(s)
I20260620 23:36:19.758486 6527 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:36:19.759455 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 6527
I20260620 23:36:19.759708 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/master-0/wal/instance
I20260620 23:36:19.760533 388 external_mini_cluster.cc:1428] Setting key eda17b7f02909888ab915c069dabea3f
I20260620 23:36:19.763587 6594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 23:36:19 dist-test-slave-tcgx krb5kdc[6514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 23:36:19.776935 6594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d4ee78c582c4c9c9ae36386fc5afcaf: Bootstrap starting.
I20260620 23:36:19.780135 6594 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d4ee78c582c4c9c9ae36386fc5afcaf: Neither blocks nor log segments found. Creating new log.
I20260620 23:36:19.781191 6594 log.cc:824] T 00000000000000000000000000000000 P 9d4ee78c582c4c9c9ae36386fc5afcaf: Log is configured to *not* fsync() on all Append() calls
I20260620 23:36:19.783910 6597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:19.762751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39342 (local address 127.0.97.62:33681)
0620 23:36:19.763604 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:19.763616 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:19.763655 (+ 39us) server_negotiation.cc:407] Connection header received
0620 23:36:19.765270 (+ 1615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:19.765293 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:19.765671 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:19.766021 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:19.767846 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:19.769267 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:19.771105 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:19.771822 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:19.771960 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:19.776681 (+ 4721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:19.776689 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:19.776705 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:19.776747 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:19.779820 (+ 3073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:19.780585 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:19.780591 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:19.780598 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:19.780751 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:19.781153 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:19.781157 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:19.781159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:19.781740 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:19.782057 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:19.782484 (+ 427us) server_negotiation.cc:299] Negotiation successful
0620 23:36:19.782895 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":428,"threads_started":1}
I20260620 23:36:19.784282 6594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d4ee78c582c4c9c9ae36386fc5afcaf: No bootstrap required, opened a new log
I20260620 23:36:19.792214 6594 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d4ee78c582c4c9c9ae36386fc5afcaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33681 } }
I20260620 23:36:19.792651 6594 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d4ee78c582c4c9c9ae36386fc5afcaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:36:19.792763 6594 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d4ee78c582c4c9c9ae36386fc5afcaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d4ee78c582c4c9c9ae36386fc5afcaf, State: Initialized, Role: FOLLOWER
I20260620 23:36:19.793380 6594 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d4ee78c582c4c9c9ae36386fc5afcaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33681 } }
I20260620 23:36:19.793637 6594 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d4ee78c582c4c9c9ae36386fc5afcaf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:36:19.793781 6594 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d4ee78c582c4c9c9ae36386fc5afcaf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:36:19.793959 6594 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d4ee78c582c4c9c9ae36386fc5afcaf [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:36:19.796569 6594 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d4ee78c582c4c9c9ae36386fc5afcaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33681 } }
I20260620 23:36:19.797116 6594 leader_election.cc:302] T 00000000000000000000000000000000 P 9d4ee78c582c4c9c9ae36386fc5afcaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d4ee78c582c4c9c9ae36386fc5afcaf; no voters:
I20260620 23:36:19.797725 6594 leader_election.cc:288] T 00000000000000000000000000000000 P 9d4ee78c582c4c9c9ae36386fc5afcaf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:36:19.797874 6599 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d4ee78c582c4c9c9ae36386fc5afcaf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:36:19.798784 6599 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d4ee78c582c4c9c9ae36386fc5afcaf [term 1 LEADER]: Becoming Leader. State: Replica: 9d4ee78c582c4c9c9ae36386fc5afcaf, State: Running, Role: LEADER
I20260620 23:36:19.799379 6599 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d4ee78c582c4c9c9ae36386fc5afcaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33681 } }
I20260620 23:36:19.799937 6594 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d4ee78c582c4c9c9ae36386fc5afcaf [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:36:19.802364 6601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d4ee78c582c4c9c9ae36386fc5afcaf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d4ee78c582c4c9c9ae36386fc5afcaf. Latest consensus state: current_term: 1 leader_uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33681 } } }
I20260620 23:36:19.802732 6601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d4ee78c582c4c9c9ae36386fc5afcaf [sys.catalog]: This master's current role is: LEADER
I20260620 23:36:19.802464 6600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d4ee78c582c4c9c9ae36386fc5afcaf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33681 } } }
I20260620 23:36:19.802882 6600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d4ee78c582c4c9c9ae36386fc5afcaf [sys.catalog]: This master's current role is: LEADER
I20260620 23:36:19.808063 6605 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:36:19.814075 6605 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:36:19.822736 6605 catalog_manager.cc:1347] Generated new cluster ID: 7fb67d48eb2a4aebad7304834a960bf3
I20260620 23:36:19.822856 6605 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:36:19.846453 6605 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:36:19.848106 6605 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:36:19.859406 6605 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d4ee78c582c4c9c9ae36386fc5afcaf: Generated new TSK 0
I20260620 23:36:19.860423 6605 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260620 23:36:19.910820 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33681
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:36:20.246671 6620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:36:20.247057 6620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:36:20.247172 6620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:36:20.254527 6620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:36:20.254644 6620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:36:20.254758 6620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 23:36:20.264906 6620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33681
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:36:20.266235 6620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:36:20.267529 6620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:36:20.280882 6625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:20.280856 6626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:20.281243 6628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:36:20.282055 6620 server_base.cc:1034] running on GCE node
I20260620 23:36:20.282836 6620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:36:20.284001 6620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:36:20.285211 6620 hybrid_clock.cc:648] HybridClock initialized: now 1781998580285166 us; error 38 us; skew 500 ppm
I20260620 23:36:20.285578 6620 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:36:20 dist-test-slave-tcgx krb5kdc[6514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998580, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:36:20.288885 6620 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260620 23:36:20.290483 6620 webserver.cc:460] Webserver started at http://127.0.97.1:38127/ using document root <none> and password file <none>
I20260620 23:36:20.291285 6620 fs_manager.cc:359] Metadata directory not provided
I20260620 23:36:20.291427 6620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:36:20.291738 6620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:36:20.294720 6620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-0/data/instance:
uuid: "c6c5446de0824443bba78a92059589a9"
format_stamp: "Formatted at 2026-06-20 23:36:20 on dist-test-slave-tcgx"
server_key: "3971d5da368e8bfcd40f27f8d2ef2eea"
server_key_iv: "82979bc661e9188ed5b080cba82601a6"
server_key_version: "encryptionkey@0"
I20260620 23:36:20.295662 6620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-0/wal/instance:
uuid: "c6c5446de0824443bba78a92059589a9"
format_stamp: "Formatted at 2026-06-20 23:36:20 on dist-test-slave-tcgx"
server_key: "3971d5da368e8bfcd40f27f8d2ef2eea"
server_key_iv: "82979bc661e9188ed5b080cba82601a6"
server_key_version: "encryptionkey@0"
I20260620 23:36:20.301314 6620 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:36:20.304955 6635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:20.306366 6620 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:36:20.306552 6620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-0/wal
uuid: "c6c5446de0824443bba78a92059589a9"
format_stamp: "Formatted at 2026-06-20 23:36:20 on dist-test-slave-tcgx"
server_key: "3971d5da368e8bfcd40f27f8d2ef2eea"
server_key_iv: "82979bc661e9188ed5b080cba82601a6"
server_key_version: "encryptionkey@0"
I20260620 23:36:20.306763 6620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:36:20.321453 6620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:36:20.326190 6620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:36:20.326445 6620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:36:20.327468 6620 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:36:20.328804 6620 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:36:20.328878 6620 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:20.328959 6620 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:36:20.329005 6620 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:20.361120 6620 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45847
I20260620 23:36:20.361179 6747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45847 every 8 connection(s)
I20260620 23:36:20.362581 6620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
I20260620 23:36:20.372460 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 6620
I20260620 23:36:20.372695 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-0/wal/instance
I20260620 23:36:20.373330 388 external_mini_cluster.cc:1428] Setting key 135bfff01ca4a1d6fe250dd2f8c504c0
Jun 20 23:36:20 dist-test-slave-tcgx krb5kdc[6514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998580, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 23:36:20.383937 6750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:20.365519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55981 (local address 127.0.97.62:33681)
0620 23:36:20.366014 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:20.366019 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:20.366910 (+ 891us) server_negotiation.cc:407] Connection header received
0620 23:36:20.368857 (+ 1947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:20.368863 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:20.368971 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:20.369092 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:20.370707 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:20.371494 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:20.373233 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:20.373702 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:20.373839 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:20.377675 (+ 3836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:20.377680 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:20.377684 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:20.377717 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:20.380541 (+ 2824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:20.381363 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:20.381369 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:20.381374 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:20.381542 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:20.382088 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:20.382092 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:20.382094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:20.382361 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:20.382478 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:20.383498 (+ 1020us) server_negotiation.cc:299] Negotiation successful
0620 23:36:20.383682 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":270,"threads_started":1}
I20260620 23:36:20.385571 6751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:20.365940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33681 (local address 127.0.97.1:55981)
0620 23:36:20.366722 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:20.366762 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:20.368549 (+ 1787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:20.369343 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:20.369351 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:20.369895 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:20.370514 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:20.370531 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:20.371667 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:20.371671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:20.373076 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:20.373088 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:20.373954 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:20.373963 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:20.374307 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:20.375105 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:20.375138 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:20.377410 (+ 2272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:36:20.380756 (+ 3346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:20.380764 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:20.380781 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:20.381225 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:20.381678 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:20.381682 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:20.381684 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:20.381973 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:20.382490 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:36:20.382502 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:36:20.383049 (+ 547us) client_negotiation.cc:769] Sending connection context
0620 23:36:20.383383 (+ 334us) client_negotiation.cc:240] Negotiation successful
0620 23:36:20.384012 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":310,"threads_started":1}
I20260620 23:36:20.387420 6748 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33681
I20260620 23:36:20.387789 6748 heartbeater.cc:464] Registering TS with master...
I20260620 23:36:20.388558 6748 heartbeater.cc:511] Master 127.0.97.62:33681 requested a full tablet report, sending...
I20260620 23:36:20.391072 6559 ts_manager.cc:194] Registered new tserver with Master: c6c5446de0824443bba78a92059589a9 (127.0.97.1:45847)
I20260620 23:36:20.393378 6559 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:55981
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260620 23:36:20.422611 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33681
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:36:20.756837 6754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:36:20.757238 6754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:36:20.757408 6754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:36:20.768944 6754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:36:20.769091 6754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:36:20.769268 6754 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 23:36:20.783160 6754 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33681
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:36:20.784554 6754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:36:20.785835 6754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:36:20.799584 6759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:20.799592 6762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:20.799645 6760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:36:20.800331 6754 server_base.cc:1034] running on GCE node
I20260620 23:36:20.801005 6754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:36:20.802042 6754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:36:20.803248 6754 hybrid_clock.cc:648] HybridClock initialized: now 1781998580803187 us; error 47 us; skew 500 ppm
I20260620 23:36:20.803618 6754 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:36:20 dist-test-slave-tcgx krb5kdc[6514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998580, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:36:20.807047 6754 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260620 23:36:20.808676 6754 webserver.cc:460] Webserver started at http://127.0.97.2:42549/ using document root <none> and password file <none>
I20260620 23:36:20.809482 6754 fs_manager.cc:359] Metadata directory not provided
I20260620 23:36:20.809633 6754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:36:20.809949 6754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:36:20.813751 6754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-1/data/instance:
uuid: "4290f95cc39945379be798778eae2e18"
format_stamp: "Formatted at 2026-06-20 23:36:20 on dist-test-slave-tcgx"
server_key: "91f133bc51b205ad1ca2f634e49b3e92"
server_key_iv: "a37887d76534f4bc3a3b20ad5d1894c9"
server_key_version: "encryptionkey@0"
I20260620 23:36:20.814625 6754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-1/wal/instance:
uuid: "4290f95cc39945379be798778eae2e18"
format_stamp: "Formatted at 2026-06-20 23:36:20 on dist-test-slave-tcgx"
server_key: "91f133bc51b205ad1ca2f634e49b3e92"
server_key_iv: "a37887d76534f4bc3a3b20ad5d1894c9"
server_key_version: "encryptionkey@0"
I20260620 23:36:20.820369 6754 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 23:36:20.824316 6769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:20.825901 6754 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:36:20.826148 6754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-1/wal
uuid: "4290f95cc39945379be798778eae2e18"
format_stamp: "Formatted at 2026-06-20 23:36:20 on dist-test-slave-tcgx"
server_key: "91f133bc51b205ad1ca2f634e49b3e92"
server_key_iv: "a37887d76534f4bc3a3b20ad5d1894c9"
server_key_version: "encryptionkey@0"
I20260620 23:36:20.826433 6754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:36:20.864491 6754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:36:20.868670 6754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:36:20.868991 6754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:36:20.869971 6754 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:36:20.871320 6754 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:36:20.871433 6754 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:20.871589 6754 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:36:20.871663 6754 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:20.904932 6754 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42859
I20260620 23:36:20.904989 6881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42859 every 8 connection(s)
I20260620 23:36:20.906255 6754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
I20260620 23:36:20.914778 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 6754
I20260620 23:36:20.914999 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-1/wal/instance
I20260620 23:36:20.915446 388 external_mini_cluster.cc:1428] Setting key bbdb19967b982f873688dc1eceb114b8
Jun 20 23:36:20 dist-test-slave-tcgx krb5kdc[6514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998580, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 23:36:20.928870 6884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:20.908974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40387 (local address 127.0.97.62:33681)
0620 23:36:20.909452 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:20.909457 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:20.910664 (+ 1207us) server_negotiation.cc:407] Connection header received
0620 23:36:20.912627 (+ 1963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:20.912633 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:20.912732 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:20.912850 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:20.914535 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:20.915442 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:20.917377 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:20.917888 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:20.917991 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:20.922455 (+ 4464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:20.922461 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:20.922465 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:20.922502 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:20.925440 (+ 2938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:20.926271 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:20.926279 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:20.926289 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:20.926422 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:20.926902 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:20.926910 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:20.926914 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:20.927265 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:20.927435 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:20.928205 (+ 770us) server_negotiation.cc:299] Negotiation successful
0620 23:36:20.928441 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":221,"threads_started":1}
I20260620 23:36:20.930073 6885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:20.909648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33681 (local address 127.0.97.2:40387)
0620 23:36:20.910381 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:20.910508 (+ 127us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:20.912330 (+ 1822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:20.913112 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:20.913121 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:20.913696 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:20.914287 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:20.914304 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:20.915615 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:20.915621 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:20.917209 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:20.917226 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:20.918142 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:20.918151 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:20.918489 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:20.919303 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:20.919340 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:20.922188 (+ 2848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:36:20.925610 (+ 3422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:20.925621 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:20.925641 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:20.926126 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:20.926551 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:20.926556 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:20.926560 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:20.926780 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:20.927405 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:36:20.927414 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:36:20.927812 (+ 398us) client_negotiation.cc:769] Sending connection context
0620 23:36:20.928109 (+ 297us) client_negotiation.cc:240] Negotiation successful
0620 23:36:20.928732 (+ 623us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":291,"threads_started":1}
I20260620 23:36:20.931818 6882 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33681
I20260620 23:36:20.932176 6882 heartbeater.cc:464] Registering TS with master...
I20260620 23:36:20.933166 6882 heartbeater.cc:511] Master 127.0.97.62:33681 requested a full tablet report, sending...
I20260620 23:36:20.934919 6559 ts_manager.cc:194] Registered new tserver with Master: 4290f95cc39945379be798778eae2e18 (127.0.97.2:42859)
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 23:36:20.935806 6559 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:40387
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260620 23:36:20.962877 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33681
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:36:21.301177 6888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:36:21.301465 6888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:36:21.301558 6888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:36:21.309043 6888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:36:21.309170 6888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:36:21.309283 6888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 23:36:21.319844 6888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33681
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:36:21.321266 6888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:36:21.322726 6888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:36:21.336593 6896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:21.336611 6894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:21.336748 6893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:36:21.337327 6888 server_base.cc:1034] running on GCE node
I20260620 23:36:21.338083 6888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:36:21.339075 6888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:36:21.340319 6888 hybrid_clock.cc:648] HybridClock initialized: now 1781998581340255 us; error 71 us; skew 500 ppm
I20260620 23:36:21.340679 6888 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:36:21 dist-test-slave-tcgx krb5kdc[6514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998581, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:36:21.347184 6888 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260620 23:36:21.348739 6888 webserver.cc:460] Webserver started at http://127.0.97.3:44257/ using document root <none> and password file <none>
I20260620 23:36:21.349462 6888 fs_manager.cc:359] Metadata directory not provided
I20260620 23:36:21.349570 6888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:36:21.349828 6888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:36:21.352936 6888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-2/data/instance:
uuid: "7fa1b50d6220465ab5e6409556e6ee7e"
format_stamp: "Formatted at 2026-06-20 23:36:21 on dist-test-slave-tcgx"
server_key: "49829f1c7fb45416d851bacd0b1116ca"
server_key_iv: "29a3960ccffbd149120a7789c2189ae3"
server_key_version: "encryptionkey@0"
I20260620 23:36:21.353721 6888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-2/wal/instance:
uuid: "7fa1b50d6220465ab5e6409556e6ee7e"
format_stamp: "Formatted at 2026-06-20 23:36:21 on dist-test-slave-tcgx"
server_key: "49829f1c7fb45416d851bacd0b1116ca"
server_key_iv: "29a3960ccffbd149120a7789c2189ae3"
server_key_version: "encryptionkey@0"
I20260620 23:36:21.359347 6888 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260620 23:36:21.363307 6903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:21.364866 6888 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:36:21.365046 6888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-2/wal
uuid: "7fa1b50d6220465ab5e6409556e6ee7e"
format_stamp: "Formatted at 2026-06-20 23:36:21 on dist-test-slave-tcgx"
server_key: "49829f1c7fb45416d851bacd0b1116ca"
server_key_iv: "29a3960ccffbd149120a7789c2189ae3"
server_key_version: "encryptionkey@0"
I20260620 23:36:21.365265 6888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:36:21.397079 6748 heartbeater.cc:503] Master 127.0.97.62:33681 was elected leader, sending a full tablet report...
I20260620 23:36:21.403726 6888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:36:21.408512 6888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:36:21.408838 6888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:36:21.409950 6888 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:36:21.411440 6888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:36:21.411548 6888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:21.411705 6888 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:36:21.411777 6888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:21.450218 6888 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36767
I20260620 23:36:21.450313 7015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36767 every 8 connection(s)
I20260620 23:36:21.451632 6888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
I20260620 23:36:21.455229 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 6888
I20260620 23:36:21.455453 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-2/wal/instance
I20260620 23:36:21.456072 388 external_mini_cluster.cc:1428] Setting key 63a8b536559e7e3cf27b90e7213b3ce0
Jun 20 23:36:21 dist-test-slave-tcgx krb5kdc[6514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998581, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 23:36:21.472872 7018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:21.454530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46077 (local address 127.0.97.62:33681)
0620 23:36:21.455076 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:21.455081 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:21.456169 (+ 1088us) server_negotiation.cc:407] Connection header received
0620 23:36:21.458286 (+ 2117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:21.458292 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:21.458399 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:21.458539 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:21.460230 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:21.461001 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:21.462576 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:21.463072 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:21.463215 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:21.466714 (+ 3499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:21.466720 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:21.466725 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:21.466767 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:21.469296 (+ 2529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:21.470255 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:21.470261 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:21.470264 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:21.470424 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:21.470870 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:21.470875 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:21.470877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:21.471305 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:21.471451 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:21.472409 (+ 958us) server_negotiation.cc:299] Negotiation successful
0620 23:36:21.472596 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":249,"threads_started":1}
I20260620 23:36:21.473894 7019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:21.455155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33681 (local address 127.0.97.3:46077)
0620 23:36:21.455983 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:21.456046 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:21.457990 (+ 1944us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:21.458778 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:21.458787 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:21.459469 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:21.460057 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:21.460075 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:21.461159 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:21.461165 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:21.462423 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:21.462435 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:21.463310 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:21.463318 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:21.463551 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:21.464266 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:21.464301 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:21.466484 (+ 2183us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:36:21.469469 (+ 2985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:21.469479 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:21.469538 (+ 59us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:21.470070 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:21.470559 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:21.470563 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:21.470567 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:21.470777 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:21.471436 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:36:21.471443 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:36:21.472008 (+ 565us) client_negotiation.cc:769] Sending connection context
0620 23:36:21.472313 (+ 305us) client_negotiation.cc:240] Negotiation successful
0620 23:36:21.472738 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":366,"threads_started":1}
I20260620 23:36:21.475361 7016 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33681
I20260620 23:36:21.475704 7016 heartbeater.cc:464] Registering TS with master...
I20260620 23:36:21.476480 7016 heartbeater.cc:511] Master 127.0.97.62:33681 requested a full tablet report, sending...
I20260620 23:36:21.478181 6559 ts_manager.cc:194] Registered new tserver with Master: 7fa1b50d6220465ab5e6409556e6ee7e (127.0.97.3:36767)
I20260620 23:36:21.479063 6559 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:46077
I20260620 23:36:21.484587 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/krb5kdc/oryx_127.0.97.61.keytab.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/krb5kdc/oryx_127.0.97.61.keytab.
I20260620 23:36:21.535303 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:39051
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33681,127.0.97.61:39051
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:36:21.875315 7022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:36:21.875677 7022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:36:21.875841 7022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:36:21.883416 7022 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:36:21.883577 7022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:36:21.883692 7022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:36:21.883795 7022 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:36:21.883870 7022 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:36:21.894307 7022 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33681,127.0.97.61:39051
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:39051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
W20260620 23:36:21.894814 7022 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.97.62:33681,127.0.97.61:39051'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 23:36:21.895941 7022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:36:21.897362 7022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:36:21.909854 7028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:21.909929 7030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:21.910326 7027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:36:21.911222 7022 server_base.cc:1034] running on GCE node
I20260620 23:36:21.912058 7022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:36:21.913630 7022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:36:21.914881 7022 hybrid_clock.cc:648] HybridClock initialized: now 1781998581914793 us; error 70 us; skew 500 ppm
I20260620 23:36:21.915242 7022 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:36:21 dist-test-slave-tcgx krb5kdc[6514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998581, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:36:21.918696 7022 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260620 23:36:21.920456 7022 webserver.cc:460] Webserver started at http://127.0.97.61:40883/ using document root <none> and password file <none>
I20260620 23:36:21.921200 7022 fs_manager.cc:359] Metadata directory not provided
I20260620 23:36:21.921296 7022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:36:21.921599 7022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:36:21.924957 7022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/master-1/data/instance:
uuid: "84fa2fff77be49b098f7d05dd4ec1a4a"
format_stamp: "Formatted at 2026-06-20 23:36:21 on dist-test-slave-tcgx"
server_key: "ec2084d764a692ca99826e7381513a3a"
server_key_iv: "cb7fd365a6ca293c0ca23cbd0a5e95b4"
server_key_version: "encryptionkey@0"
I20260620 23:36:21.925791 7022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/master-1/wal/instance:
uuid: "84fa2fff77be49b098f7d05dd4ec1a4a"
format_stamp: "Formatted at 2026-06-20 23:36:21 on dist-test-slave-tcgx"
server_key: "ec2084d764a692ca99826e7381513a3a"
server_key_iv: "cb7fd365a6ca293c0ca23cbd0a5e95b4"
server_key_version: "encryptionkey@0"
I20260620 23:36:21.931890 7022 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.007s
I20260620 23:36:21.935568 7037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:21.937103 7022 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260620 23:36:21.937253 7022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/master-1/wal
uuid: "84fa2fff77be49b098f7d05dd4ec1a4a"
format_stamp: "Formatted at 2026-06-20 23:36:21 on dist-test-slave-tcgx"
server_key: "ec2084d764a692ca99826e7381513a3a"
server_key_iv: "cb7fd365a6ca293c0ca23cbd0a5e95b4"
server_key_version: "encryptionkey@0"
I20260620 23:36:21.937453 7022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:36:21.938567 6882 heartbeater.cc:503] Master 127.0.97.62:33681 was elected leader, sending a full tablet report...
I20260620 23:36:21.955442 7022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:36:21.960210 7022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:36:21.960492 7022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:36:21.979305 7022 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:39051
I20260620 23:36:21.979344 7088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:39051 every 8 connection(s)
I20260620 23:36:21.980993 7022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/master-1/data/info.pb
I20260620 23:36:21.984989 7089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:36:21.986811 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 7022
I20260620 23:36:21.994582 7089 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33681 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:36:21 dist-test-slave-tcgx krb5kdc[6514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Jun 20 23:36:22 dist-test-slave-tcgx krb5kdc[6514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998581, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 23:36:22.011276 7092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:21.989337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56428 (local address 127.0.97.61:39051)
0620 23:36:21.990087 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:21.990099 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:21.990147 (+ 48us) server_negotiation.cc:407] Connection header received
0620 23:36:21.991818 (+ 1671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:21.991843 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:21.992281 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:21.992731 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:21.994222 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:21.995789 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:21.997040 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:21.998414 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:21.998610 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:22.002861 (+ 4251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:22.002869 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:22.002881 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:22.003018 (+ 137us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:22.006191 (+ 3173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:22.007068 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:22.007077 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:22.007093 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:22.007245 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:22.007699 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:22.007705 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:22.007708 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:22.008539 (+ 831us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:22.008833 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:22.009254 (+ 421us) server_negotiation.cc:299] Negotiation successful
0620 23:36:22.009753 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"spinlock_wait_cycles":15104,"thread_start_us":325,"threads_started":1}
I20260620 23:36:22.012408 7094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:21.997248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39350 (local address 127.0.97.62:33681)
0620 23:36:21.998501 (+ 1253us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:21.998506 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:21.998521 (+ 15us) server_negotiation.cc:407] Connection header received
0620 23:36:21.998568 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:21.998572 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:21.998635 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:21.998758 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:21.999891 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:22.000629 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:22.002003 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:22.002468 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:22.002603 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:22.005978 (+ 3375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:22.005983 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:22.005986 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:22.006022 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:22.009354 (+ 3332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:22.010090 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:22.010096 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:22.010099 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:22.010230 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:22.010671 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:22.010676 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:22.010679 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:22.011121 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:22.011278 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:22.011823 (+ 545us) server_negotiation.cc:299] Negotiation successful
0620 23:36:22.012050 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1154,"thread_start_us":1064,"threads_started":1}
I20260620 23:36:22.012487 7093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:21.997194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33681 (local address 127.0.0.1:39350)
0620 23:36:21.997934 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:21.997984 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:21.998125 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:21.998823 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:21.998830 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:21.999470 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:21.999718 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:21.999726 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:22.000765 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:22.000772 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:22.001862 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:22.001873 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:22.002699 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:22.002707 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:22.002869 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:22.003563 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:22.003589 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:22.005783 (+ 2194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:36:22.009526 (+ 3743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:22.009536 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:22.009543 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:22.009969 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:22.010355 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:22.010358 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:22.010360 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:22.010552 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:22.011270 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:36:22.011276 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:36:22.011470 (+ 194us) client_negotiation.cc:769] Sending connection context
0620 23:36:22.011732 (+ 262us) client_negotiation.cc:240] Negotiation successful
0620 23:36:22.012076 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":335,"threads_started":1}
I20260620 23:36:22.014199 7089 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39051 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:36:22 dist-test-slave-tcgx krb5kdc[6514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998581, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260620 23:36:22.026551 7093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:22.015073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39051 (local address 127.0.0.1:56434)
0620 23:36:22.015260 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:22.015277 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:22.015362 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:22.015628 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:22.015631 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:22.015935 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:22.016085 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:22.016093 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:22.017151 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:22.017155 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:22.018020 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:22.018031 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:22.018726 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:22.018731 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:22.018868 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:22.019471 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:22.019488 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:22.021392 (+ 1904us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:36:22.024295 (+ 2903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:22.024299 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:22.024302 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:22.024730 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:22.025102 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:22.025105 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:22.025108 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:22.025260 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:22.025835 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:36:22.025839 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:36:22.025974 (+ 135us) client_negotiation.cc:769] Sending connection context
0620 23:36:22.026093 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 23:36:22.026305 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 23:36:22.026732 7092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:22.015252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56434 (local address 127.0.97.61:39051)
0620 23:36:22.015394 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:22.015398 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:22.015413 (+ 15us) server_negotiation.cc:407] Connection header received
0620 23:36:22.015452 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:22.015455 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:22.015511 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:22.015586 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:22.016206 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:22.016943 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:22.018149 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:22.018584 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:22.018680 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:22.021560 (+ 2880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:22.021565 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:22.021568 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:22.021593 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:22.024155 (+ 2562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:22.024853 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:22.024857 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:22.024860 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:22.025002 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:22.025377 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:22.025381 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:22.025384 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:22.025724 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:22.025837 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:22.026207 (+ 370us) server_negotiation.cc:299] Negotiation successful
0620 23:36:22.026426 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":49}
I20260620 23:36:22.031755 7089 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84fa2fff77be49b098f7d05dd4ec1a4a: Bootstrap starting.
I20260620 23:36:22.034890 7089 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84fa2fff77be49b098f7d05dd4ec1a4a: Neither blocks nor log segments found. Creating new log.
I20260620 23:36:22.035983 7089 log.cc:824] T 00000000000000000000000000000000 P 84fa2fff77be49b098f7d05dd4ec1a4a: Log is configured to *not* fsync() on all Append() calls
I20260620 23:36:22.038592 7089 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84fa2fff77be49b098f7d05dd4ec1a4a: No bootstrap required, opened a new log
I20260620 23:36:22.045881 7089 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84fa2fff77be49b098f7d05dd4ec1a4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33681 } } peers { permanent_uuid: "84fa2fff77be49b098f7d05dd4ec1a4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39051 } }
I20260620 23:36:22.046242 7089 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84fa2fff77be49b098f7d05dd4ec1a4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:36:22.046343 7089 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84fa2fff77be49b098f7d05dd4ec1a4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84fa2fff77be49b098f7d05dd4ec1a4a, State: Initialized, Role: FOLLOWER
I20260620 23:36:22.047029 7089 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84fa2fff77be49b098f7d05dd4ec1a4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33681 } } peers { permanent_uuid: "84fa2fff77be49b098f7d05dd4ec1a4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39051 } }
I20260620 23:36:22.047853 7096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84fa2fff77be49b098f7d05dd4ec1a4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33681 } } peers { permanent_uuid: "84fa2fff77be49b098f7d05dd4ec1a4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39051 } } }
I20260620 23:36:22.048130 7096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84fa2fff77be49b098f7d05dd4ec1a4a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:36:22.048884 7089 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84fa2fff77be49b098f7d05dd4ec1a4a [sys.catalog]: configured and running, proceeding with master startup.
W20260620 23:36:22.057662 7107 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 84fa2fff77be49b098f7d05dd4ec1a4a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:36:22.057814 7107 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:36:22.060272 7022 master_runner.cc:428] Detected that this master 84fa2fff77be49b098f7d05dd4ec1a4a is joining an existing cluster
I20260620 23:36:22.060364 7022 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.61:39051
I20260620 23:36:22.088502 7094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:22.064193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39358 (local address 127.0.97.62:33681)
0620 23:36:22.064389 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:22.064398 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:22.071747 (+ 7349us) server_negotiation.cc:407] Connection header received
0620 23:36:22.071943 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:22.071948 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:22.072047 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:22.072145 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:22.073762 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:22.074911 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:22.076665 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:22.077388 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:22.077541 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:22.080857 (+ 3316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:22.080866 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:22.080874 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:22.080920 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:22.084926 (+ 4006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:22.085827 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:22.085837 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:22.085843 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:22.086012 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:22.086503 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:22.086511 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:22.086516 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:22.086906 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:22.087069 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:22.087751 (+ 682us) server_negotiation.cc:299] Negotiation successful
0620 23:36:22.088021 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 23:36:22.089783 7114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:22.064244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33681 (local address 127.0.0.1:39358)
0620 23:36:22.071613 (+ 7369us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:22.071639 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:22.071809 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:22.072310 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:22.072318 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:22.073132 (+ 814us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:22.073575 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:22.073592 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:22.075148 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:22.075156 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:22.076503 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:22.076515 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:22.077845 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:22.077853 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:22.078068 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:22.078743 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:22.078772 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:22.080627 (+ 1855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:36:22.085102 (+ 4475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:22.085114 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:22.085121 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:22.085683 (+ 562us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:22.086169 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:22.086176 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:22.086182 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:22.086360 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:22.087177 (+ 817us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:36:22.087183 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:36:22.087426 (+ 243us) client_negotiation.cc:769] Sending connection context
0620 23:36:22.088788 (+ 1362us) client_negotiation.cc:240] Negotiation successful
0620 23:36:22.089163 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7181,"thread_start_us":16139,"threads_started":1}
WARNING: no policy specified for oryx/127.0.97.60@KRBTEST.COM; defaulting to no policy
I20260620 23:36:22.096954 7113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:22.064875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39051 (local address 127.0.0.1:56438)
0620 23:36:22.079811 (+ 14936us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:22.079836 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:22.080699 (+ 863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:22.081182 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:22.081186 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:22.081835 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:22.082162 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:22.082181 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:22.083657 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:22.083667 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:22.085325 (+ 1658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:22.085339 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:22.086624 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:22.086632 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:22.086816 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:22.087476 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:22.087504 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:22.088535 (+ 1031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:36:22.093863 (+ 5328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:22.093867 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:22.093871 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:22.094449 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:22.094863 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:22.094867 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:22.094870 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:22.095063 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:22.095761 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:36:22.095767 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:36:22.095901 (+ 134us) client_negotiation.cc:769] Sending connection context
0620 23:36:22.096046 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 23:36:22.096384 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14777,"thread_start_us":7029,"threads_started":1}
I20260620 23:36:22.096982 7092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:22.064229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56438 (local address 127.0.97.61:39051)
0620 23:36:22.064548 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:22.064553 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:22.080372 (+ 15819us) server_negotiation.cc:407] Connection header received
0620 23:36:22.080839 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:22.080844 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:22.080909 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:22.080971 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:22.082327 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:22.083447 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:22.085473 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:22.086199 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:22.086316 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:22.089028 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:22.089034 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:22.089039 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:22.089074 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:22.093700 (+ 4626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:22.094571 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:22.094576 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:22.094579 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:22.094736 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:22.095183 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:22.095188 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:22.095190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:22.095569 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:22.095683 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:22.096214 (+ 531us) server_negotiation.cc:299] Negotiation successful
0620 23:36:22.096576 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":189}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.60@KRBTEST.COM" created.
Jun 20 23:36:22 dist-test-slave-tcgx krb5kdc[6514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998579, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260620 23:36:22.119479 7116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:22.097014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39051 (local address 127.0.0.1:56442)
0620 23:36:22.097764 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:22.097814 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:22.097985 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:22.103173 (+ 5188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:22.103185 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:22.104267 (+ 1082us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:22.104656 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:22.104675 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:22.106569 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:22.106577 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:22.108309 (+ 1732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:22.108337 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:22.109293 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:22.109299 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:22.109524 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:22.110360 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:22.110379 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:22.112843 (+ 2464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:36:22.116485 (+ 3642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:22.116498 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:22.116507 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:22.116967 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:22.117421 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:22.117426 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:22.117430 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:22.117600 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:22.118205 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:36:22.118214 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:36:22.118417 (+ 203us) client_negotiation.cc:769] Sending connection context
0620 23:36:22.118641 (+ 224us) client_negotiation.cc:240] Negotiation successful
0620 23:36:22.119047 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":356,"threads_started":1}
I20260620 23:36:22.120100 7092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:22.097494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56442 (local address 127.0.97.61:39051)
0620 23:36:22.098421 (+ 927us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:22.098427 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:22.098442 (+ 15us) server_negotiation.cc:407] Connection header received
0620 23:36:22.098526 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:22.098531 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:22.098651 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:22.098752 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:22.104895 (+ 6143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:22.106365 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:22.108523 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:22.109085 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:22.109260 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:22.113092 (+ 3832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:22.113098 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:22.113101 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:22.113163 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:22.116304 (+ 3141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:22.117130 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:22.117136 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:22.117139 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:22.117287 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:22.117727 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:22.117732 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:22.117736 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:22.118032 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:22.118233 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:22.119151 (+ 918us) server_negotiation.cc:299] Negotiation successful
0620 23:36:22.119409 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":818,"thread_start_us":7989,"threads_started":1}
I20260620 23:36:22.121899 6559 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.61:39051: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "84fa2fff77be49b098f7d05dd4ec1a4a"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.61"
port: 39051
}
attrs {
promote: true
}
}
dest_uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf"
cas_config_opid_index: -1
I20260620 23:36:22.122925 6559 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d4ee78c582c4c9c9ae36386fc5afcaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33681 } } peers { permanent_uuid: "84fa2fff77be49b098f7d05dd4ec1a4a" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39051 } attrs { promote: true } }
I20260620 23:36:22.126909 7120 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9d4ee78c582c4c9c9ae36386fc5afcaf [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 84fa2fff77be49b098f7d05dd4ec1a4a (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33681 } } peers { permanent_uuid: "84fa2fff77be49b098f7d05dd4ec1a4a" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39051 } attrs { promote: true } } }
I20260620 23:36:22.127995 7064 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84fa2fff77be49b098f7d05dd4ec1a4a [term 0 FOLLOWER]: Advancing to term 1
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/krb5kdc/oryx_127.0.97.60.keytab.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/krb5kdc/oryx_127.0.97.60.keytab.
I20260620 23:36:22.133023 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:33735
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33681,127.0.97.61:39051,127.0.97.60:33735
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:36:22.133140 7120 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.61:39051
I20260620 23:36:22.133656 7120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d4ee78c582c4c9c9ae36386fc5afcaf [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33681 } } peers { permanent_uuid: "84fa2fff77be49b098f7d05dd4ec1a4a" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39051 } attrs { promote: true } } }
I20260620 23:36:22.133927 7120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d4ee78c582c4c9c9ae36386fc5afcaf [sys.catalog]: This master's current role is: LEADER
I20260620 23:36:22.134052 7022 master.cc:537] Master@127.0.97.61:39051 shutting down...
I20260620 23:36:22.144098 7064 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 84fa2fff77be49b098f7d05dd4ec1a4a [term 1 FOLLOWER]: Refusing update from remote peer 9d4ee78c582c4c9c9ae36386fc5afcaf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 23:36:22.145648 7121 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9d4ee78c582c4c9c9ae36386fc5afcaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "84fa2fff77be49b098f7d05dd4ec1a4a" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39051 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:36:22.148711 7022 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 84fa2fff77be49b098f7d05dd4ec1a4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:36:22.149171 7022 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 84fa2fff77be49b098f7d05dd4ec1a4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:36:22.149381 7022 tablet_replica.cc:330] T 00000000000000000000000000000000 P 84fa2fff77be49b098f7d05dd4ec1a4a: stopping tablet replica
W20260620 23:36:22.149466 6546 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260620 23:36:22.157301 7022 master.cc:552] Master@127.0.97.61:39051 shutdown complete.
I20260620 23:36:22.157399 7022 master_runner.cc:305] Clearing existing system tablet
I20260620 23:36:22.157830 7022 fs_manager.cc:359] Metadata directory not provided
I20260620 23:36:22.157927 7022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
W20260620 23:36:22.160876 6546 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9d4ee78c582c4c9c9ae36386fc5afcaf -> Peer 84fa2fff77be49b098f7d05dd4ec1a4a (127.0.97.61:39051): Couldn't send request to peer 84fa2fff77be49b098f7d05dd4ec1a4a. Status: Network error: recv got EOF from unknown peer (error 108). This is attempt 1: this message will repeat every 5th retry.
I20260620 23:36:22.162036 7022 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260620 23:36:22.163837 7126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:22.164228 7022 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:36:22.164521 7022 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 23:36:22.164606 7022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/master-1/wal
uuid: "84fa2fff77be49b098f7d05dd4ec1a4a"
format_stamp: "Formatted at 2026-06-20 23:36:21 on dist-test-slave-tcgx"
server_key: "ec2084d764a692ca99826e7381513a3a"
server_key_iv: "cb7fd365a6ca293c0ca23cbd0a5e95b4"
server_key_version: "encryptionkey@0"
I20260620 23:36:22.169819 7022 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 84fa2fff77be49b098f7d05dd4ec1a4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:36:22.177240 7022 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 84fa2fff77be49b098f7d05dd4ec1a4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 23:36:22.177450 7022 log.cc:1196] T 00000000000000000000000000000000 P 84fa2fff77be49b098f7d05dd4ec1a4a: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 23:36:22.177917 7022 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 84fa2fff77be49b098f7d05dd4ec1a4a: Deleting consensus metadata
I20260620 23:36:22.178248 7022 master_runner.cc:315] Copying system tablet from 127.0.97.62:33681
I20260620 23:36:22.180869 7022 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 84fa2fff77be49b098f7d05dd4ec1a4a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:33681
I20260620 23:36:22.198187 7094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:22.182467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39362 (local address 127.0.97.62:33681)
0620 23:36:22.182665 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:22.182672 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:22.183352 (+ 680us) server_negotiation.cc:407] Connection header received
0620 23:36:22.183544 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:22.183550 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:22.183660 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:22.183779 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:22.184923 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:22.186019 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:22.187950 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:22.188611 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:22.188761 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:22.191206 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:22.191214 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:22.191221 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:22.191273 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:22.195017 (+ 3744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:22.195925 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:22.195931 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:22.195935 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:22.196098 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:22.196541 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:22.196546 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:22.196549 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:22.196914 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:22.197071 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:22.197543 (+ 472us) server_negotiation.cc:299] Negotiation successful
0620 23:36:22.197779 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 23:36:22.198230 7132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:22.181862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33681 (local address 127.0.0.1:39362)
0620 23:36:22.183206 (+ 1344us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:22.183228 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:22.183398 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:22.183826 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:22.183832 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:22.184403 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:22.184740 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:22.184752 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:22.186197 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:22.186203 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:22.187779 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:22.187795 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:22.188801 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:22.188808 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:22.188990 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:22.189821 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:22.189846 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:22.190971 (+ 1125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:36:22.195199 (+ 4228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:22.195205 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:22.195209 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:22.195766 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:22.196227 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:22.196232 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:22.196236 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:22.196408 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:22.197056 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:36:22.197065 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:36:22.197255 (+ 190us) client_negotiation.cc:769] Sending connection context
0620 23:36:22.197408 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 23:36:22.197770 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1194,"thread_start_us":986,"threads_started":1}
I20260620 23:36:22.199620 6579 tablet_copy_service.cc:140] P 9d4ee78c582c4c9c9ae36386fc5afcaf: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 84fa2fff77be49b098f7d05dd4ec1a4a ({username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:39362)
I20260620 23:36:22.199956 6579 tablet_copy_service.cc:161] P 9d4ee78c582c4c9c9ae36386fc5afcaf: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 84fa2fff77be49b098f7d05dd4ec1a4a at {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:39362: session id = 84fa2fff77be49b098f7d05dd4ec1a4a-00000000000000000000000000000000
I20260620 23:36:22.205493 6579 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 9d4ee78c582c4c9c9ae36386fc5afcaf: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 23:36:22.208118 7022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 23:36:22.215351 7022 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 84fa2fff77be49b098f7d05dd4ec1a4a: tablet copy: Starting download of 0 data blocks...
I20260620 23:36:22.215691 7022 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 84fa2fff77be49b098f7d05dd4ec1a4a: tablet copy: Starting download of 1 WAL segments...
I20260620 23:36:22.218751 7022 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 84fa2fff77be49b098f7d05dd4ec1a4a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 23:36:22.223488 6579 tablet_copy_service.cc:342] P 9d4ee78c582c4c9c9ae36386fc5afcaf: Request end of tablet copy session 84fa2fff77be49b098f7d05dd4ec1a4a-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:39362
I20260620 23:36:22.223770 6579 tablet_copy_service.cc:433] P 9d4ee78c582c4c9c9ae36386fc5afcaf: ending tablet copy session 84fa2fff77be49b098f7d05dd4ec1a4a-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 84fa2fff77be49b098f7d05dd4ec1a4a
I20260620 23:36:22.236756 7022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:36:22.239735 7133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:22.240891 7134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:36:22.248443 7022 server_base.cc:1034] running on GCE node
W20260620 23:36:22.248445 7136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:36:22.249213 7022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:36:22.249882 7022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:36:22.251088 7022 hybrid_clock.cc:648] HybridClock initialized: now 1781998582251024 us; error 37 us; skew 500 ppm
I20260620 23:36:22.251330 7022 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:36:22 dist-test-slave-tcgx krb5kdc[6514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998582, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:36:22.253600 7022 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260620 23:36:22.254982 7022 webserver.cc:460] Webserver started at http://127.0.97.61:42891/ using document root <none> and password file <none>
I20260620 23:36:22.255350 7022 fs_manager.cc:359] Metadata directory not provided
I20260620 23:36:22.255435 7022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:36:22.258626 7022 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260620 23:36:22.260761 7143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:22.261179 7022 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:36:22.261390 7022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/master-1/wal
uuid: "84fa2fff77be49b098f7d05dd4ec1a4a"
format_stamp: "Formatted at 2026-06-20 23:36:21 on dist-test-slave-tcgx"
server_key: "ec2084d764a692ca99826e7381513a3a"
server_key_iv: "cb7fd365a6ca293c0ca23cbd0a5e95b4"
server_key_version: "encryptionkey@0"
I20260620 23:36:22.261575 7022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:36:22.276877 7022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:36:22.281155 7022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:36:22.305352 7022 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:39051
I20260620 23:36:22.305397 7194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:39051 every 8 connection(s)
I20260620 23:36:22.306319 7022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/master-1/data/info.pb
I20260620 23:36:22.311981 7195 sys_catalog.cc:263] Verifying existing consensus state
I20260620 23:36:22.316258 7195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84fa2fff77be49b098f7d05dd4ec1a4a: Bootstrap starting.
I20260620 23:36:22.334359 7195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84fa2fff77be49b098f7d05dd4ec1a4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:36:22.334892 7195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84fa2fff77be49b098f7d05dd4ec1a4a: Bootstrap complete.
I20260620 23:36:22.336074 7195 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84fa2fff77be49b098f7d05dd4ec1a4a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33681 } } peers { permanent_uuid: "84fa2fff77be49b098f7d05dd4ec1a4a" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39051 } attrs { promote: true } }
I20260620 23:36:22.336345 7195 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84fa2fff77be49b098f7d05dd4ec1a4a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 84fa2fff77be49b098f7d05dd4ec1a4a, State: Initialized, Role: LEARNER
I20260620 23:36:22.336628 7195 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84fa2fff77be49b098f7d05dd4ec1a4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33681 } } peers { permanent_uuid: "84fa2fff77be49b098f7d05dd4ec1a4a" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39051 } attrs { promote: true } }
I20260620 23:36:22.337850 7195 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84fa2fff77be49b098f7d05dd4ec1a4a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:36:22.337656 7198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84fa2fff77be49b098f7d05dd4ec1a4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33681 } } peers { permanent_uuid: "84fa2fff77be49b098f7d05dd4ec1a4a" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39051 } attrs { promote: true } } }
I20260620 23:36:22.338193 7198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84fa2fff77be49b098f7d05dd4ec1a4a [sys.catalog]: This master's current role is: LEARNER
I20260620 23:36:22.346596 7209 catalog_manager.cc:1259] Loaded cluster ID: 7fb67d48eb2a4aebad7304834a960bf3
I20260620 23:36:22.346731 7209 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 84fa2fff77be49b098f7d05dd4ec1a4a: loading cluster ID for follower catalog manager: success
I20260620 23:36:22.350677 7209 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 84fa2fff77be49b098f7d05dd4ec1a4a: acquiring CA information for follower catalog manager: success
I20260620 23:36:22.353518 7209 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 84fa2fff77be49b098f7d05dd4ec1a4a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 23:36:22.481856 7016 heartbeater.cc:503] Master 127.0.97.62:33681 was elected leader, sending a full tablet report...
W20260620 23:36:22.545743 7123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:36:22.546049 7123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:36:22.546139 7123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:36:22.554565 7123 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:36:22.554713 7123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:36:22.554764 7123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:36:22.554829 7123 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:36:22.554878 7123 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:36:22.566262 7123 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33681,127.0.97.61:39051,127.0.97.60:33735
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:33735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:36:22.567834 7123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:36:22.569306 7123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:36:22.582762 7214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:22.582808 7215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:22.582703 7217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:36:22.583981 7123 server_base.cc:1034] running on GCE node
I20260620 23:36:22.584924 7123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:36:22.586674 7123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:36:22.587939 7123 hybrid_clock.cc:648] HybridClock initialized: now 1781998582587864 us; error 69 us; skew 500 ppm
I20260620 23:36:22.588346 7123 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:36:22 dist-test-slave-tcgx krb5kdc[6514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998582, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:36:22.591742 7123 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260620 23:36:22.593317 7123 webserver.cc:460] Webserver started at http://127.0.97.60:33745/ using document root <none> and password file <none>
I20260620 23:36:22.594053 7123 fs_manager.cc:359] Metadata directory not provided
I20260620 23:36:22.594202 7123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:36:22.594502 7123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:36:22.598059 7123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/master-2/data/instance:
uuid: "4b672bfe633a443591fc2dd3ca7277b1"
format_stamp: "Formatted at 2026-06-20 23:36:22 on dist-test-slave-tcgx"
server_key: "5b23547ca833a310a00087e9f7619f09"
server_key_iv: "dec20cb5a7f37e5d6261da245e230479"
server_key_version: "encryptionkey@0"
I20260620 23:36:22.598826 7123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/master-2/wal/instance:
uuid: "4b672bfe633a443591fc2dd3ca7277b1"
format_stamp: "Formatted at 2026-06-20 23:36:22 on dist-test-slave-tcgx"
server_key: "5b23547ca833a310a00087e9f7619f09"
server_key_iv: "dec20cb5a7f37e5d6261da245e230479"
server_key_version: "encryptionkey@0"
I20260620 23:36:22.604117 7123 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 23:36:22.607599 7224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:22.609122 7123 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:36:22.609292 7123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/master-2/wal
uuid: "4b672bfe633a443591fc2dd3ca7277b1"
format_stamp: "Formatted at 2026-06-20 23:36:22 on dist-test-slave-tcgx"
server_key: "5b23547ca833a310a00087e9f7619f09"
server_key_iv: "dec20cb5a7f37e5d6261da245e230479"
server_key_version: "encryptionkey@0"
I20260620 23:36:22.609483 7123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:36:22.634122 7123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:36:22.639361 7123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:36:22.639652 7123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:36:22.659637 7123 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:33735
I20260620 23:36:22.659694 7275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:33735 every 8 connection(s)
I20260620 23:36:22.661070 7123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/master-2/data/info.pb
I20260620 23:36:22.665417 7276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:36:22.669723 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 7123
I20260620 23:36:22.674090 7276 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33681 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:36:22 dist-test-slave-tcgx krb5kdc[6514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Jun 20 23:36:22 dist-test-slave-tcgx krb5kdc[6514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998582, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 23:36:22.695710 7094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:22.678123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39376 (local address 127.0.97.62:33681)
0620 23:36:22.678341 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:22.678347 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:22.678910 (+ 563us) server_negotiation.cc:407] Connection header received
0620 23:36:22.679130 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:22.679134 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:22.679194 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:22.679274 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:22.680861 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:22.681943 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:22.684743 (+ 2800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:22.685196 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:22.685345 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:22.689373 (+ 4028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:22.689378 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:22.689381 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:22.689413 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:22.692437 (+ 3024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:22.693310 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:22.693318 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:22.693324 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:22.693499 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:22.694015 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:22.694020 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:22.694022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:22.694289 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:22.694468 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:22.695121 (+ 653us) server_negotiation.cc:299] Negotiation successful
0620 23:36:22.695470 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260620 23:36:22.696910 7279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:22.672208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34126 (local address 127.0.97.60:33735)
0620 23:36:22.673013 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:22.673027 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:22.673080 (+ 53us) server_negotiation.cc:407] Connection header received
0620 23:36:22.674709 (+ 1629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:22.674732 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:22.675257 (+ 525us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:22.676086 (+ 829us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:22.678105 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:22.680460 (+ 2355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:22.682714 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:22.684048 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:22.684249 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:22.688876 (+ 4627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:22.688887 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:22.688905 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:22.688959 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:22.691980 (+ 3021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:22.692824 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:22.692833 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:22.692836 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:22.693020 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:22.693573 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:22.693579 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:22.693584 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:22.694258 (+ 674us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:22.694506 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:22.694900 (+ 394us) server_negotiation.cc:299] Negotiation successful
0620 23:36:22.695424 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":114,"server-negotiator.queue_time_us":595,"thread_start_us":372,"threads_started":1}
I20260620 23:36:22.696930 7280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:22.677874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33681 (local address 127.0.0.1:39376)
0620 23:36:22.678747 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:22.678802 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:22.679008 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:22.679380 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:22.679389 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:22.680324 (+ 935us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:22.680714 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:22.680724 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:22.682109 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:22.682113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:22.684595 (+ 2482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:22.684605 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:22.685438 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:22.685446 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:22.685615 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:22.686238 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:22.686262 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:22.689151 (+ 2889us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:36:22.692571 (+ 3420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:22.692579 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:22.692590 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:22.693158 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:22.693600 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:22.693604 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:22.693607 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:22.693863 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:22.694456 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:36:22.694465 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:36:22.694660 (+ 195us) client_negotiation.cc:769] Sending connection context
0620 23:36:22.694959 (+ 299us) client_negotiation.cc:240] Negotiation successful
0620 23:36:22.695519 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"mutex_wait_us":179,"spinlock_wait_cycles":25984,"thread_start_us":401,"threads_started":1}
I20260620 23:36:22.698911 7276 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39051 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:36:22 dist-test-slave-tcgx krb5kdc[6514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998582, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260620 23:36:22.713891 7280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:22.699781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39051 (local address 127.0.0.1:56452)
0620 23:36:22.699967 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:22.699984 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:22.700098 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:22.700777 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:22.700782 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:22.701276 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:22.701507 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:22.701519 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:22.703444 (+ 1925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:22.703448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:22.704728 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:22.704740 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:22.705380 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:22.705384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:22.705573 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:22.706136 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:22.706153 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:22.708176 (+ 2023us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:36:22.711569 (+ 3393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:22.711573 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:22.711576 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:22.711946 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:22.712300 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:22.712304 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:22.712306 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:22.712430 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:22.712975 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:36:22.712978 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:36:22.713166 (+ 188us) client_negotiation.cc:769] Sending connection context
0620 23:36:22.713311 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 23:36:22.713639 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 23:36:22.713922 7281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:22.699889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56452 (local address 127.0.97.61:39051)
0620 23:36:22.700452 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:22.700458 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:22.700476 (+ 18us) server_negotiation.cc:407] Connection header received
0620 23:36:22.700550 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:22.700554 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:22.700612 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:22.700699 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:22.701713 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:22.703270 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:22.704866 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:22.705255 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:22.705354 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:22.708397 (+ 3043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:22.708402 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:22.708405 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:22.708437 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:22.711414 (+ 2977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:22.712066 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:22.712070 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:22.712073 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:22.712201 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:22.712566 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:22.712570 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:22.712574 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:22.712867 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:22.712997 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:22.713418 (+ 421us) server_negotiation.cc:299] Negotiation successful
0620 23:36:22.713589 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":298,"threads_started":1}
I20260620 23:36:22.715206 7276 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33735 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:36:22 dist-test-slave-tcgx krb5kdc[6514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998582, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260620 23:36:22.728716 7279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:22.715962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34142 (local address 127.0.97.60:33735)
0620 23:36:22.716110 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:22.716114 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:22.716139 (+ 25us) server_negotiation.cc:407] Connection header received
0620 23:36:22.716271 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:22.716275 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:22.716328 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:22.716402 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:22.717064 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:22.717740 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:22.718900 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:22.719777 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:22.719879 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:22.723451 (+ 3572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:22.723455 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:22.723458 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:22.723486 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:22.726202 (+ 2716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:22.726802 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:22.726810 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:22.726814 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:22.726920 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:22.727358 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:22.727363 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:22.727366 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:22.727619 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:22.727758 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:22.728075 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 23:36:22.728399 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":136,"server-negotiator.queue_time_us":58}
I20260620 23:36:22.728754 7281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:22.722778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56454 (local address 127.0.97.61:39051)
0620 23:36:22.722999 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:22.723005 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:22.723202 (+ 197us) server_negotiation.cc:407] Connection header received
0620 23:36:22.723398 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:22.723404 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:22.723503 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:22.723620 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:36:22.724100 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:22.725296 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:22.726850 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:22.727796 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:22.727950 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:22.728193 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:22.728370 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 23:36:22.728504 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 23:36:22.728721 7280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:22.715906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33735 (local address 127.0.0.1:34142)
0620 23:36:22.716069 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:22.716089 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:22.716184 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:22.716437 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:22.716440 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:22.716779 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:22.716917 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:22.716925 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:22.717871 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:22.717875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:22.718755 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:22.718765 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:22.719907 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:22.719912 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:22.720080 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:22.720652 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:22.720668 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:22.723272 (+ 2604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:36:22.726326 (+ 3054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:22.726334 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:22.726338 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:22.726690 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:22.727091 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:22.727097 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:22.727104 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:22.727247 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:22.727733 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:36:22.727738 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:36:22.727868 (+ 130us) client_negotiation.cc:769] Sending connection context
0620 23:36:22.728000 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 23:36:22.728274 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 23:36:22.729292 7283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:22.722595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39051 (local address 127.0.0.1:56454)
0620 23:36:22.723101 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:22.723120 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:22.723262 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:22.723665 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:22.723671 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:22.723694 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:36:22.723889 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:22.723943 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:22.725438 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:22.725442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:22.726692 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:22.726703 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:22.727969 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:22.727975 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:22.728156 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:22.728224 (+ 68us) client_negotiation.cc:769] Sending connection context
0620 23:36:22.728344 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 23:36:22.728485 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":220,"threads_started":1}
I20260620 23:36:22.730338 7170 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 84fa2fff77be49b098f7d05dd4ec1a4a [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260620 23:36:22.731065 7198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84fa2fff77be49b098f7d05dd4ec1a4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d4ee78c582c4c9c9ae36386fc5afcaf. Latest consensus state: current_term: 1 leader_uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33681 } } peers { permanent_uuid: "84fa2fff77be49b098f7d05dd4ec1a4a" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39051 } attrs { promote: true } } }
I20260620 23:36:22.731339 7198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84fa2fff77be49b098f7d05dd4ec1a4a [sys.catalog]: This master's current role is: LEARNER
I20260620 23:36:22.734697 7276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b672bfe633a443591fc2dd3ca7277b1: Bootstrap starting.
I20260620 23:36:22.738417 7276 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b672bfe633a443591fc2dd3ca7277b1: Neither blocks nor log segments found. Creating new log.
I20260620 23:36:22.739620 7276 log.cc:824] T 00000000000000000000000000000000 P 4b672bfe633a443591fc2dd3ca7277b1: Log is configured to *not* fsync() on all Append() calls
I20260620 23:36:22.742766 7276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b672bfe633a443591fc2dd3ca7277b1: No bootstrap required, opened a new log
I20260620 23:36:22.751029 7276 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b672bfe633a443591fc2dd3ca7277b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33681 } } peers { permanent_uuid: "84fa2fff77be49b098f7d05dd4ec1a4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39051 } } peers { permanent_uuid: "4b672bfe633a443591fc2dd3ca7277b1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33735 } }
I20260620 23:36:22.751521 7276 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b672bfe633a443591fc2dd3ca7277b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:36:22.751691 7276 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b672bfe633a443591fc2dd3ca7277b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b672bfe633a443591fc2dd3ca7277b1, State: Initialized, Role: FOLLOWER
I20260620 23:36:22.752399 7276 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b672bfe633a443591fc2dd3ca7277b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33681 } } peers { permanent_uuid: "84fa2fff77be49b098f7d05dd4ec1a4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39051 } } peers { permanent_uuid: "4b672bfe633a443591fc2dd3ca7277b1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33735 } }
I20260620 23:36:22.753391 7285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b672bfe633a443591fc2dd3ca7277b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33681 } } peers { permanent_uuid: "84fa2fff77be49b098f7d05dd4ec1a4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39051 } } peers { permanent_uuid: "4b672bfe633a443591fc2dd3ca7277b1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33735 } } }
I20260620 23:36:22.753788 7285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b672bfe633a443591fc2dd3ca7277b1 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:36:22.754674 7276 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b672bfe633a443591fc2dd3ca7277b1 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 23:36:22.765462 7296 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4b672bfe633a443591fc2dd3ca7277b1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:36:22.765659 7296 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:36:22.770171 7123 master_runner.cc:428] Detected that this master 4b672bfe633a443591fc2dd3ca7277b1 is joining an existing cluster
I20260620 23:36:22.770356 7123 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.60:33735
I20260620 23:36:22.802698 7279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:22.775691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34146 (local address 127.0.97.60:33735)
0620 23:36:22.775884 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:22.775890 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:22.781386 (+ 5496us) server_negotiation.cc:407] Connection header received
0620 23:36:22.781848 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:22.781857 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:22.781948 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:22.782029 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:22.784165 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:22.785653 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:22.788347 (+ 2694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:22.789324 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:22.789454 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:22.792865 (+ 3411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:22.792877 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:22.792884 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:22.792926 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:22.799041 (+ 6115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:22.799965 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:22.799970 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:22.799977 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:22.800180 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:22.800556 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:22.800560 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:22.800564 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:22.801369 (+ 805us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:22.801514 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:22.802025 (+ 511us) server_negotiation.cc:299] Negotiation successful
0620 23:36:22.802260 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 23:36:22.803687 7304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:22.776447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33735 (local address 127.0.0.1:34146)
0620 23:36:22.781237 (+ 4790us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:22.781264 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:22.781651 (+ 387us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:22.782257 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:22.782264 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:22.783568 (+ 1304us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:22.783955 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:22.783976 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:22.785943 (+ 1967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:22.785951 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:22.788154 (+ 2203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:22.788171 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:22.789876 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:22.789886 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:22.790138 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:22.791032 (+ 894us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:22.791062 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:22.792298 (+ 1236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:36:22.799188 (+ 6890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:22.799193 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:22.799198 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:22.799776 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:22.800302 (+ 526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:22.800307 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:22.800310 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:22.800460 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:22.801577 (+ 1117us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:36:22.801584 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:36:22.801756 (+ 172us) client_negotiation.cc:769] Sending connection context
0620 23:36:22.802903 (+ 1147us) client_negotiation.cc:240] Negotiation successful
0620 23:36:22.803216 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4586,"thread_start_us":18512,"threads_started":1}
I20260620 23:36:22.804029 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 6620
I20260620 23:36:22.807902 7094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:22.775539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39384 (local address 127.0.97.62:33681)
0620 23:36:22.775739 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:22.775745 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:22.791677 (+ 15932us) server_negotiation.cc:407] Connection header received
0620 23:36:22.791937 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:22.791945 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:22.792041 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:22.792112 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:22.794207 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:22.795447 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:22.797293 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:22.797917 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:22.798045 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:22.800454 (+ 2409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:22.800462 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:22.800467 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:22.800507 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:22.804035 (+ 3528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:22.805380 (+ 1345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:22.805387 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:22.805393 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:22.805558 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:22.806051 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:22.806056 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:22.806061 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:22.806433 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:22.806555 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:22.807290 (+ 735us) server_negotiation.cc:299] Negotiation successful
0620 23:36:22.807541 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 23:36:22.809427 7303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:22.776540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33681 (local address 127.0.0.1:39384)
0620 23:36:22.791552 (+ 15012us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:22.791577 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:22.791791 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:22.792331 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:22.792338 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:22.793572 (+ 1234us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:22.794007 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:22.794027 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:22.795654 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:22.795661 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:22.797140 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:22.797151 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:22.798134 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:22.798140 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:22.798311 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:22.799018 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:22.799040 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:22.800227 (+ 1187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:36:22.804730 (+ 4503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:22.804736 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:22.804739 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:22.805239 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:22.805717 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:22.805725 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:22.805733 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:22.805911 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:22.806660 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:36:22.806666 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:36:22.806895 (+ 229us) client_negotiation.cc:769] Sending connection context
0620 23:36:22.808150 (+ 1255us) client_negotiation.cc:240] Negotiation successful
0620 23:36:22.808480 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14823,"thread_start_us":14406,"threads_started":1}
I20260620 23:36:22.813315 7281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:22.775880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56468 (local address 127.0.97.61:39051)
0620 23:36:22.776128 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:22.776134 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:22.795411 (+ 19277us) server_negotiation.cc:407] Connection header received
0620 23:36:22.795557 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:22.795563 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:22.795648 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:22.795741 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:22.797380 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:22.798372 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:22.800447 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:22.801540 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:22.801653 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:22.804417 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:22.804423 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:22.804428 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:22.804485 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:22.809815 (+ 5330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:22.810605 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:22.810611 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:22.810615 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:22.810776 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:22.811261 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:22.811268 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:22.811272 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:22.811596 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:22.811746 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:22.812679 (+ 933us) server_negotiation.cc:299] Negotiation successful
0620 23:36:22.812915 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260620 23:36:22.814553 7302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:22.777480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39051 (local address 127.0.0.1:56468)
0620 23:36:22.795271 (+ 17791us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:22.795292 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:22.795452 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:22.795921 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:22.795926 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:22.796700 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:22.797049 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:22.797064 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:22.798501 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:22.798507 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:22.800290 (+ 1783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:22.800300 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:22.801896 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:22.801904 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:22.802084 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:22.802571 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:22.802590 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:22.804206 (+ 1616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:36:22.809967 (+ 5761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:22.809973 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:22.809976 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:22.810465 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:22.810902 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:22.810907 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:22.810910 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:22.811129 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:22.811806 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:36:22.811811 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:36:22.812014 (+ 203us) client_negotiation.cc:769] Sending connection context
0620 23:36:22.812157 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 23:36:22.812467 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17644,"thread_start_us":3924,"threads_started":1}
Jun 20 23:36:22 dist-test-slave-tcgx krb5kdc[6514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
Jun 20 23:36:22 dist-test-slave-tcgx krb5kdc[6514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998579, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260620 23:36:22.826035 7305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:22.805527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55318 (local address 127.0.97.1:45847)
0620 23:36:22.806249 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:22.806258 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:22.806290 (+ 32us) server_negotiation.cc:407] Connection header received
0620 23:36:22.806356 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:22.806365 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:22.806657 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:22.808543 (+ 1886us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:22.808799 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:22.810994 (+ 2195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:22.813188 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:22.814010 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:22.814212 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:22.819257 (+ 5045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:22.819269 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:22.819284 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:22.819329 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:22.823094 (+ 3765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:22.823813 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:22.823828 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:22.823831 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:22.823954 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:22.824353 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:22.824360 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:22.824363 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:22.824719 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:22.825151 (+ 432us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:22.825350 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 23:36:22.825612 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":1306,"threads_started":1}
I20260620 23:36:22.827615 6743 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:36:22.833552 7283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:22.815696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33735 (local address 127.0.0.1:34150)
0620 23:36:22.815890 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:22.815910 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:22.816012 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:22.816711 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:22.816718 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:22.817205 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:22.817405 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:22.817420 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:22.818878 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:22.818884 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:22.820392 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:22.820407 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:22.821946 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:22.821954 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:22.822153 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:22.822752 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:22.822777 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:22.825761 (+ 2984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:36:22.830878 (+ 5117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:22.830886 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:22.830892 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:22.831474 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:22.831890 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:22.831896 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:22.831900 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:22.832038 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:22.832587 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:36:22.832592 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:36:22.832744 (+ 152us) client_negotiation.cc:769] Sending connection context
0620 23:36:22.832885 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 23:36:22.833186 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 23:36:22.833555 7279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:22.816082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34150 (local address 127.0.97.60:33735)
0620 23:36:22.816276 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:22.816285 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:22.816306 (+ 21us) server_negotiation.cc:407] Connection header received
0620 23:36:22.816399 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:22.816407 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:22.816546 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:22.816643 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:22.817568 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:22.818693 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:22.820587 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:22.821769 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:22.821930 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:22.826009 (+ 4079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:22.826017 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:22.826024 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:22.826072 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:22.830642 (+ 4570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:22.831614 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:22.831619 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:22.831623 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:22.831762 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:22.832166 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:22.832170 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:22.832174 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:22.832463 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:22.832624 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:22.832987 (+ 363us) server_negotiation.cc:299] Negotiation successful
0620 23:36:22.833212 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 23:36:22.835345 6559 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.60:33735: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "4b672bfe633a443591fc2dd3ca7277b1"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.60"
port: 33735
}
attrs {
promote: true
}
}
dest_uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf"
cas_config_opid_index: 5
I20260620 23:36:22.836231 6559 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d4ee78c582c4c9c9ae36386fc5afcaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33681 } } peers { permanent_uuid: "84fa2fff77be49b098f7d05dd4ec1a4a" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39051 } attrs { promote: true } } peers { permanent_uuid: "4b672bfe633a443591fc2dd3ca7277b1" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 33735 } attrs { promote: true } }
I20260620 23:36:22.838182 7282 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9d4ee78c582c4c9c9ae36386fc5afcaf [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 4b672bfe633a443591fc2dd3ca7277b1 (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33681 } } peers { permanent_uuid: "84fa2fff77be49b098f7d05dd4ec1a4a" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39051 } attrs { promote: true } } peers { permanent_uuid: "4b672bfe633a443591fc2dd3ca7277b1" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 33735 } attrs { promote: true } } }
I20260620 23:36:22.844530 7170 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 84fa2fff77be49b098f7d05dd4ec1a4a [term 1 LEARNER]: Refusing update from remote peer 9d4ee78c582c4c9c9ae36386fc5afcaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 23:36:22.846577 7251 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b672bfe633a443591fc2dd3ca7277b1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:36:22.848459 7307 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9d4ee78c582c4c9c9ae36386fc5afcaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "84fa2fff77be49b098f7d05dd4ec1a4a" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39051 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 23:36:22.850965 7251 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4b672bfe633a443591fc2dd3ca7277b1 [term 1 FOLLOWER]: Refusing update from remote peer 9d4ee78c582c4c9c9ae36386fc5afcaf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 23:36:22.855989 7170 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 84fa2fff77be49b098f7d05dd4ec1a4a [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 4b672bfe633a443591fc2dd3ca7277b1 (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33681 } } peers { permanent_uuid: "84fa2fff77be49b098f7d05dd4ec1a4a" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39051 } attrs { promote: true } } peers { permanent_uuid: "4b672bfe633a443591fc2dd3ca7277b1" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 33735 } attrs { promote: true } } }
I20260620 23:36:22.858421 7310 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9d4ee78c582c4c9c9ae36386fc5afcaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b672bfe633a443591fc2dd3ca7277b1" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 33735 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:36:22.859014 7282 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.60:33735
I20260620 23:36:22.859694 7123 master.cc:537] Master@127.0.97.60:33735 shutting down...
I20260620 23:36:22.861720 7198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84fa2fff77be49b098f7d05dd4ec1a4a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33681 } } peers { permanent_uuid: "84fa2fff77be49b098f7d05dd4ec1a4a" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39051 } attrs { promote: true } } peers { permanent_uuid: "4b672bfe633a443591fc2dd3ca7277b1" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 33735 } attrs { promote: true } } }
I20260620 23:36:22.861968 7198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84fa2fff77be49b098f7d05dd4ec1a4a [sys.catalog]: This master's current role is: LEARNER
I20260620 23:36:22.874100 7311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d4ee78c582c4c9c9ae36386fc5afcaf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33681 } } peers { permanent_uuid: "84fa2fff77be49b098f7d05dd4ec1a4a" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39051 } attrs { promote: true } } peers { permanent_uuid: "4b672bfe633a443591fc2dd3ca7277b1" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 33735 } attrs { promote: true } } }
I20260620 23:36:22.874531 7312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d4ee78c582c4c9c9ae36386fc5afcaf [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33681 } } peers { permanent_uuid: "84fa2fff77be49b098f7d05dd4ec1a4a" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 39051 } attrs { promote: true } } peers { permanent_uuid: "4b672bfe633a443591fc2dd3ca7277b1" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 33735 } attrs { promote: true } } }
I20260620 23:36:22.874876 7312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d4ee78c582c4c9c9ae36386fc5afcaf [sys.catalog]: This master's current role is: LEADER
I20260620 23:36:22.874053 7282 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 9d4ee78c582c4c9c9ae36386fc5afcaf: attempting to promote NON_VOTER 84fa2fff77be49b098f7d05dd4ec1a4a to VOTER
I20260620 23:36:22.876068 7282 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d4ee78c582c4c9c9ae36386fc5afcaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33681 } } peers { permanent_uuid: "84fa2fff77be49b098f7d05dd4ec1a4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39051 } attrs { promote: false } } peers { permanent_uuid: "4b672bfe633a443591fc2dd3ca7277b1" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 33735 } attrs { promote: true } }
I20260620 23:36:22.874804 7311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d4ee78c582c4c9c9ae36386fc5afcaf [sys.catalog]: This master's current role is: LEADER
I20260620 23:36:22.879618 7170 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 84fa2fff77be49b098f7d05dd4ec1a4a [term 1 LEARNER]: Refusing update from remote peer 9d4ee78c582c4c9c9ae36386fc5afcaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 23:36:22.880458 7307 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9d4ee78c582c4c9c9ae36386fc5afcaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "84fa2fff77be49b098f7d05dd4ec1a4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39051 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 23:36:22.883994 7170 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 84fa2fff77be49b098f7d05dd4ec1a4a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 84fa2fff77be49b098f7d05dd4ec1a4a (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33681 } } peers { permanent_uuid: "84fa2fff77be49b098f7d05dd4ec1a4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39051 } attrs { promote: false } } peers { permanent_uuid: "4b672bfe633a443591fc2dd3ca7277b1" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 33735 } attrs { promote: true } } }
I20260620 23:36:22.884327 7312 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9d4ee78c582c4c9c9ae36386fc5afcaf [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 84fa2fff77be49b098f7d05dd4ec1a4a (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33681 } } peers { permanent_uuid: "84fa2fff77be49b098f7d05dd4ec1a4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39051 } attrs { promote: false } } peers { permanent_uuid: "4b672bfe633a443591fc2dd3ca7277b1" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 33735 } attrs { promote: true } } }
I20260620 23:36:22.888908 7198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84fa2fff77be49b098f7d05dd4ec1a4a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33681 } } peers { permanent_uuid: "84fa2fff77be49b098f7d05dd4ec1a4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39051 } attrs { promote: false } } peers { permanent_uuid: "4b672bfe633a443591fc2dd3ca7277b1" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 33735 } attrs { promote: true } } }
I20260620 23:36:22.889168 7198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84fa2fff77be49b098f7d05dd4ec1a4a [sys.catalog]: This master's current role is: FOLLOWER
W20260620 23:36:22.889467 6544 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9d4ee78c582c4c9c9ae36386fc5afcaf -> Peer 4b672bfe633a443591fc2dd3ca7277b1 (127.0.97.60:33735): Couldn't send request to peer 4b672bfe633a443591fc2dd3ca7277b1. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 23:36:22.890347 7308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d4ee78c582c4c9c9ae36386fc5afcaf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33681 } } peers { permanent_uuid: "84fa2fff77be49b098f7d05dd4ec1a4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39051 } attrs { promote: false } } peers { permanent_uuid: "4b672bfe633a443591fc2dd3ca7277b1" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 33735 } attrs { promote: true } } }
I20260620 23:36:22.890656 7308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d4ee78c582c4c9c9ae36386fc5afcaf [sys.catalog]: This master's current role is: LEADER
I20260620 23:36:22.891022 7311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d4ee78c582c4c9c9ae36386fc5afcaf [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33681 } } peers { permanent_uuid: "84fa2fff77be49b098f7d05dd4ec1a4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39051 } attrs { promote: false } } peers { permanent_uuid: "4b672bfe633a443591fc2dd3ca7277b1" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 33735 } attrs { promote: true } } }
I20260620 23:36:22.891247 7311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d4ee78c582c4c9c9ae36386fc5afcaf [sys.catalog]: This master's current role is: LEADER
W20260620 23:36:22.895262 7230 tablet.cc:2369] T 00000000000000000000000000000000 P 4b672bfe633a443591fc2dd3ca7277b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:36:22.897271 7123 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 4b672bfe633a443591fc2dd3ca7277b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:36:22.897694 7123 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 4b672bfe633a443591fc2dd3ca7277b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:36:22.897902 7123 tablet_replica.cc:330] T 00000000000000000000000000000000 P 4b672bfe633a443591fc2dd3ca7277b1: stopping tablet replica
I20260620 23:36:22.913982 7123 master.cc:552] Master@127.0.97.60:33735 shutdown complete.
I20260620 23:36:22.914168 7123 master_runner.cc:305] Clearing existing system tablet
I20260620 23:36:22.914575 7123 fs_manager.cc:359] Metadata directory not provided
I20260620 23:36:22.914712 7123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:36:22.917039 7123 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:36:22.918514 7316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:22.918901 7123 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:36:22.919186 7123 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:36:22.919288 7123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/master-2/wal
uuid: "4b672bfe633a443591fc2dd3ca7277b1"
format_stamp: "Formatted at 2026-06-20 23:36:22 on dist-test-slave-tcgx"
server_key: "5b23547ca833a310a00087e9f7619f09"
server_key_iv: "dec20cb5a7f37e5d6261da245e230479"
server_key_version: "encryptionkey@0"
I20260620 23:36:22.922713 7123 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 4b672bfe633a443591fc2dd3ca7277b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:36:22.927655 7123 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 4b672bfe633a443591fc2dd3ca7277b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 23:36:22.927862 7123 log.cc:1196] T 00000000000000000000000000000000 P 4b672bfe633a443591fc2dd3ca7277b1: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 23:36:22.928794 7123 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 4b672bfe633a443591fc2dd3ca7277b1: Deleting consensus metadata
I20260620 23:36:22.929071 7123 master_runner.cc:315] Copying system tablet from 127.0.97.62:33681
I20260620 23:36:22.931934 7123 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 4b672bfe633a443591fc2dd3ca7277b1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:33681
I20260620 23:36:22.946262 7094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:22.932724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39386 (local address 127.0.97.62:33681)
0620 23:36:22.933010 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:22.933020 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:22.933558 (+ 538us) server_negotiation.cc:407] Connection header received
0620 23:36:22.933791 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:22.933798 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:22.933880 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:22.933960 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:22.934875 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:22.935656 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:22.936884 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:22.937319 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:22.937460 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:22.939580 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:22.939587 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:22.939592 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:22.939655 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:22.943463 (+ 3808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:22.944160 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:22.944167 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:22.944171 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:22.944312 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:22.944734 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:22.944740 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:22.944743 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:22.945028 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:22.945186 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:22.945640 (+ 454us) server_negotiation.cc:299] Negotiation successful
0620 23:36:22.945857 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260620 23:36:22.946338 7322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:22.932702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33681 (local address 127.0.0.1:39386)
0620 23:36:22.933416 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:22.933436 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:22.933626 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:22.934125 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:22.934128 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:22.934521 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:22.934722 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:22.934732 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:22.935822 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:22.935832 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:22.936738 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:22.936754 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:22.937487 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:22.937548 (+ 61us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:22.937695 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:22.938422 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:22.938439 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:22.939383 (+ 944us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:36:22.943610 (+ 4227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:22.943616 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:22.943620 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:22.944014 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:22.944429 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:22.944434 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:22.944438 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:22.944600 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:22.945149 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:36:22.945154 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:36:22.945334 (+ 180us) client_negotiation.cc:769] Sending connection context
0620 23:36:22.945533 (+ 199us) client_negotiation.cc:240] Negotiation successful
0620 23:36:22.945861 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":453,"threads_started":1}
I20260620 23:36:22.947238 6579 tablet_copy_service.cc:140] P 9d4ee78c582c4c9c9ae36386fc5afcaf: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 4b672bfe633a443591fc2dd3ca7277b1 ({username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:39386)
I20260620 23:36:22.947414 6579 tablet_copy_service.cc:161] P 9d4ee78c582c4c9c9ae36386fc5afcaf: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 4b672bfe633a443591fc2dd3ca7277b1 at {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:39386: session id = 4b672bfe633a443591fc2dd3ca7277b1-00000000000000000000000000000000
I20260620 23:36:22.949643 6579 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 9d4ee78c582c4c9c9ae36386fc5afcaf: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 23:36:22.951361 7123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 23:36:22.956415 7123 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 4b672bfe633a443591fc2dd3ca7277b1: tablet copy: Starting download of 0 data blocks...
I20260620 23:36:22.956734 7123 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 4b672bfe633a443591fc2dd3ca7277b1: tablet copy: Starting download of 1 WAL segments...
I20260620 23:36:22.959940 7123 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 4b672bfe633a443591fc2dd3ca7277b1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 23:36:22.964645 6579 tablet_copy_service.cc:342] P 9d4ee78c582c4c9c9ae36386fc5afcaf: Request end of tablet copy session 4b672bfe633a443591fc2dd3ca7277b1-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:39386
I20260620 23:36:22.964804 6579 tablet_copy_service.cc:433] P 9d4ee78c582c4c9c9ae36386fc5afcaf: ending tablet copy session 4b672bfe633a443591fc2dd3ca7277b1-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 4b672bfe633a443591fc2dd3ca7277b1
I20260620 23:36:22.975533 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 6620
I20260620 23:36:22.977960 7123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:36:22.980651 7323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:22.984392 7324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:22.988692 7326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:36:22.997000 7123 server_base.cc:1034] running on GCE node
I20260620 23:36:22.997392 7123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:36:22.999207 7123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:36:23.000530 7123 hybrid_clock.cc:648] HybridClock initialized: now 1781998583000462 us; error 50 us; skew 500 ppm
I20260620 23:36:23.000639 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:45847
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:33681
--webserver_port=38127
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:33681,127.0.97.61:39051
--tserver_master_addrs=127.0.97.62:33681,127.0.97.61:39051,127.0.97.60:33735 with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:36:23.000931 7123 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:36:22 dist-test-slave-tcgx krb5kdc[6514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998582, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:36:23.002898 7123 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260620 23:36:23.008031 7123 webserver.cc:460] Webserver started at http://127.0.97.60:36983/ using document root <none> and password file <none>
I20260620 23:36:23.008350 7123 fs_manager.cc:359] Metadata directory not provided
I20260620 23:36:23.008481 7123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:36:23.011076 7123 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260620 23:36:23.012992 7334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:23.013417 7123 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260620 23:36:23.013578 7123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/master-2/wal
uuid: "4b672bfe633a443591fc2dd3ca7277b1"
format_stamp: "Formatted at 2026-06-20 23:36:22 on dist-test-slave-tcgx"
server_key: "5b23547ca833a310a00087e9f7619f09"
server_key_iv: "dec20cb5a7f37e5d6261da245e230479"
server_key_version: "encryptionkey@0"
I20260620 23:36:23.013768 7123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:36:23.030441 7123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:36:23.034253 7123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:36:23.062716 7123 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:33735
I20260620 23:36:23.062740 7385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:33735 every 8 connection(s)
I20260620 23:36:23.063758 7123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/master-2/data/info.pb
I20260620 23:36:23.071149 7386 sys_catalog.cc:263] Verifying existing consensus state
I20260620 23:36:23.076020 7386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b672bfe633a443591fc2dd3ca7277b1: Bootstrap starting.
I20260620 23:36:23.091858 7386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b672bfe633a443591fc2dd3ca7277b1: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:36:23.092346 7386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b672bfe633a443591fc2dd3ca7277b1: Bootstrap complete.
I20260620 23:36:23.093262 7386 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b672bfe633a443591fc2dd3ca7277b1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33681 } } peers { permanent_uuid: "84fa2fff77be49b098f7d05dd4ec1a4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39051 } attrs { promote: false } } peers { permanent_uuid: "4b672bfe633a443591fc2dd3ca7277b1" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 33735 } attrs { promote: true } }
I20260620 23:36:23.093513 7386 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b672bfe633a443591fc2dd3ca7277b1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4b672bfe633a443591fc2dd3ca7277b1, State: Initialized, Role: LEARNER
I20260620 23:36:23.093766 7386 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b672bfe633a443591fc2dd3ca7277b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33681 } } peers { permanent_uuid: "84fa2fff77be49b098f7d05dd4ec1a4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39051 } attrs { promote: false } } peers { permanent_uuid: "4b672bfe633a443591fc2dd3ca7277b1" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 33735 } attrs { promote: true } }
I20260620 23:36:23.094587 7389 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b672bfe633a443591fc2dd3ca7277b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33681 } } peers { permanent_uuid: "84fa2fff77be49b098f7d05dd4ec1a4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39051 } attrs { promote: false } } peers { permanent_uuid: "4b672bfe633a443591fc2dd3ca7277b1" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 33735 } attrs { promote: true } } }
I20260620 23:36:23.094888 7386 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b672bfe633a443591fc2dd3ca7277b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:36:23.095141 7389 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b672bfe633a443591fc2dd3ca7277b1 [sys.catalog]: This master's current role is: LEARNER
I20260620 23:36:23.103597 7400 catalog_manager.cc:1259] Loaded cluster ID: 7fb67d48eb2a4aebad7304834a960bf3
I20260620 23:36:23.103734 7400 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4b672bfe633a443591fc2dd3ca7277b1: loading cluster ID for follower catalog manager: success
I20260620 23:36:23.106280 7400 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4b672bfe633a443591fc2dd3ca7277b1: acquiring CA information for follower catalog manager: success
I20260620 23:36:23.108498 7400 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4b672bfe633a443591fc2dd3ca7277b1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 23:36:23.394905 7402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:23.387396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34156 (local address 127.0.97.60:33735)
0620 23:36:23.389468 (+ 2072us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:23.389478 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:23.389502 (+ 24us) server_negotiation.cc:407] Connection header received
0620 23:36:23.389582 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:23.389587 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:23.389670 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:23.389781 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:36:23.390262 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:23.391525 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:23.393067 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:23.393873 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:23.394032 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:23.394311 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:23.394410 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 23:36:23.394542 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1942,"thread_start_us":1778,"threads_started":1}
I20260620 23:36:23.394905 7401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:23.386968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33735 (local address 127.0.0.1:34156)
0620 23:36:23.389057 (+ 2089us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:23.389097 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:23.389260 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:23.389834 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:23.389840 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:23.389873 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:36:23.390091 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:23.390102 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:23.391681 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:23.391688 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:23.392878 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:23.392891 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:23.394074 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:23.394080 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:23.394245 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:23.394288 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 23:36:23.394380 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 23:36:23.394527 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1905,"thread_start_us":1811,"threads_started":1}
I20260620 23:36:23.395836 7361 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 4b672bfe633a443591fc2dd3ca7277b1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260620 23:36:23.396477 7389 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b672bfe633a443591fc2dd3ca7277b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d4ee78c582c4c9c9ae36386fc5afcaf. Latest consensus state: current_term: 1 leader_uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33681 } } peers { permanent_uuid: "84fa2fff77be49b098f7d05dd4ec1a4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39051 } attrs { promote: false } } peers { permanent_uuid: "4b672bfe633a443591fc2dd3ca7277b1" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 33735 } attrs { promote: true } } }
I20260620 23:36:23.396788 7389 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b672bfe633a443591fc2dd3ca7277b1 [sys.catalog]: This master's current role is: LEARNER
W20260620 23:36:23.434900 7331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:36:23.435300 7331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:36:23.435462 7331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:36:23.443994 7331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:36:23.444124 7331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:36:23.444303 7331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 23:36:23.456606 7331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:45847
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=38127
--tserver_master_addrs=127.0.97.62:33681,127.0.97.61:39051,127.0.97.60:33735
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:36:23.458025 7331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:36:23.459486 7331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:36:23.474802 7407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:23.474912 7408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:23.474903 7410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:36:23.481118 7331 server_base.cc:1034] running on GCE node
I20260620 23:36:23.481899 7331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:36:23.483110 7331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:36:23.484382 7331 hybrid_clock.cc:648] HybridClock initialized: now 1781998583484331 us; error 45 us; skew 500 ppm
I20260620 23:36:23.484876 7331 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:36:23 dist-test-slave-tcgx krb5kdc[6514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998583, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:36:23.488374 7331 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260620 23:36:23.489949 7331 webserver.cc:460] Webserver started at http://127.0.97.1:38127/ using document root <none> and password file <none>
I20260620 23:36:23.490684 7331 fs_manager.cc:359] Metadata directory not provided
I20260620 23:36:23.490795 7331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:36:23.497388 7331 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:36:23.500640 7417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:23.502372 7331 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:36:23.502602 7331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-0/wal
uuid: "c6c5446de0824443bba78a92059589a9"
format_stamp: "Formatted at 2026-06-20 23:36:20 on dist-test-slave-tcgx"
server_key: "3971d5da368e8bfcd40f27f8d2ef2eea"
server_key_iv: "82979bc661e9188ed5b080cba82601a6"
server_key_version: "encryptionkey@0"
I20260620 23:36:23.504053 7331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:36:23.534680 7331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:36:23.539659 7331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:36:23.540015 7331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:36:23.541260 7331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:36:23.543313 7331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:36:23.543445 7331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:23.543594 7331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:36:23.543697 7331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:23.581955 7331 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45847
I20260620 23:36:23.582053 7529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45847 every 8 connection(s)
I20260620 23:36:23.583524 7331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
I20260620 23:36:23.590902 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 7331
I20260620 23:36:23.591384 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 6754
Jun 20 23:36:23 dist-test-slave-tcgx krb5kdc[6514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
Jun 20 23:36:23 dist-test-slave-tcgx krb5kdc[6514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998583, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Jun 20 23:36:23 dist-test-slave-tcgx krb5kdc[6514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998583, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Jun 20 23:36:23 dist-test-slave-tcgx krb5kdc[6514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998583, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260620 23:36:23.651551 7538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:23.593222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55874 (local address 127.0.97.2:42859)
0620 23:36:23.620588 (+ 27366us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:23.620603 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:23.621919 (+ 1316us) server_negotiation.cc:407] Connection header received
0620 23:36:23.622034 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:23.622048 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:23.622458 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:23.622668 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:23.625733 (+ 3065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:23.628348 (+ 2615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:23.630452 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:23.631486 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:23.631620 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:23.639510 (+ 7890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:23.639531 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:23.639551 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:23.639600 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:23.646668 (+ 7068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:23.647925 (+ 1257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:23.647938 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:23.647942 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:23.648123 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:23.648703 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:23.648708 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:23.648712 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:23.649209 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:23.649512 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:23.650794 (+ 1282us) server_negotiation.cc:299] Negotiation successful
0620 23:36:23.651113 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27070,"thread_start_us":27386,"threads_started":1}
I20260620 23:36:23.651640 7540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:23.594365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57787 (local address 127.0.97.61:39051)
0620 23:36:23.621862 (+ 27497us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:23.621871 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:23.632554 (+ 10683us) server_negotiation.cc:407] Connection header received
0620 23:36:23.632840 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:23.632847 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:23.632958 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:23.633068 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:23.635254 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:23.636433 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:23.638621 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:23.639383 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:23.639491 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:23.644540 (+ 5049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:23.644549 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:23.644557 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:23.644607 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:23.648270 (+ 3663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:23.649121 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:23.649127 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:23.649131 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:23.649278 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:23.649740 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:23.649746 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:23.649749 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:23.650093 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:23.650220 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:23.651056 (+ 836us) server_negotiation.cc:299] Negotiation successful
0620 23:36:23.651267 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27316,"thread_start_us":27512,"threads_started":1}
I20260620 23:36:23.651669 7402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:23.586787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57585 (local address 127.0.97.60:33735)
0620 23:36:23.587006 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:23.587012 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:23.630222 (+ 43210us) server_negotiation.cc:407] Connection header received
0620 23:36:23.630522 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:23.630531 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:23.630649 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:23.630738 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:23.633207 (+ 2469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:23.634194 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:23.636693 (+ 2499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:23.637366 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:23.637467 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:23.643618 (+ 6151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:23.643626 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:23.643630 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:23.643689 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:23.647512 (+ 3823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:23.648709 (+ 1197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:23.648715 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:23.648718 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:23.648868 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:23.649441 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:23.649447 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:23.649450 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:23.649820 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:23.649939 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:23.651055 (+ 1116us) server_negotiation.cc:299] Negotiation successful
0620 23:36:23.651312 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 23:36:23.651715 7534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:23.586785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42685 (local address 127.0.97.62:33681)
0620 23:36:23.628635 (+ 41850us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:23.628645 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:23.628678 (+ 33us) server_negotiation.cc:407] Connection header received
0620 23:36:23.629020 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:23.629029 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:23.629141 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:23.629296 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:23.631778 (+ 2482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:23.633050 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:23.636694 (+ 3644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:23.637366 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:23.637467 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:23.643201 (+ 5734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:23.643209 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:23.643214 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:23.643262 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:23.647475 (+ 4213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:23.648708 (+ 1233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:23.648714 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:23.648718 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:23.648867 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:23.649441 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:23.649447 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:23.649450 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:23.649782 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:23.649914 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:23.651052 (+ 1138us) server_negotiation.cc:299] Negotiation successful
0620 23:36:23.651302 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41654,"thread_start_us":41488,"threads_started":1}
I20260620 23:36:23.653774 6877 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:36:23.654234 7539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:23.587552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33681 (local address 127.0.97.1:42685)
0620 23:36:23.623438 (+ 35886us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:23.623531 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:23.628641 (+ 5110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:23.629589 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:23.629602 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:23.630542 (+ 940us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:23.631555 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:23.631590 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:23.633294 (+ 1704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:23.633298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:23.636511 (+ 3213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:23.636523 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:23.637897 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:23.637907 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:23.638412 (+ 505us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:23.639464 (+ 1052us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:23.639504 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:23.642850 (+ 3346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:36:23.647669 (+ 4819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:23.647680 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:23.647701 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:23.648553 (+ 852us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:23.649026 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:23.649031 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:23.649035 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:23.649316 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:23.650010 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:36:23.650019 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:36:23.650564 (+ 545us) client_negotiation.cc:769] Sending connection context
0620 23:36:23.651897 (+ 1333us) client_negotiation.cc:240] Negotiation successful
0620 23:36:23.652729 (+ 832us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35442,"mutex_wait_us":504,"spinlock_wait_cycles":156288,"thread_start_us":44476,"threads_started":1}
I20260620 23:36:23.654258 7536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:23.593768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39051 (local address 127.0.97.1:57787)
0620 23:36:23.632443 (+ 38675us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:23.632465 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:23.632668 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:23.633341 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:23.633347 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:23.634418 (+ 1071us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:23.635017 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:23.635032 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:23.636643 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:23.636651 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:23.638440 (+ 1789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:23.638456 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:23.639809 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:23.639815 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:23.640023 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:23.640714 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:23.640745 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:23.644261 (+ 3516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:36:23.648440 (+ 4179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:23.648446 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:23.648450 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:23.648969 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:23.649424 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:23.649428 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:23.649432 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:23.649596 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:23.650306 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:36:23.650313 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:36:23.650564 (+ 251us) client_negotiation.cc:769] Sending connection context
0620 23:36:23.652025 (+ 1461us) client_negotiation.cc:240] Negotiation successful
0620 23:36:23.652599 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38522,"mutex_wait_us":112,"spinlock_wait_cycles":29952,"thread_start_us":33178,"threads_started":1}
I20260620 23:36:23.654325 7535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:23.587599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33735 (local address 127.0.97.1:57585)
0620 23:36:23.630078 (+ 42479us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:23.630107 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:23.630352 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:23.631077 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:23.631085 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:23.632640 (+ 1555us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:23.633001 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:23.633013 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:23.634415 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:23.634420 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:23.636510 (+ 2090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:23.636523 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:23.637906 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:23.637911 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:23.638424 (+ 513us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:23.639464 (+ 1040us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:23.639508 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:23.643360 (+ 3852us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:36:23.647669 (+ 4309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:23.647680 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:23.647701 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:23.648553 (+ 852us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:23.649026 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:23.649031 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:23.649035 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:23.649307 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:23.650010 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:36:23.650019 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:36:23.650569 (+ 550us) client_negotiation.cc:769] Sending connection context
0620 23:36:23.651923 (+ 1354us) client_negotiation.cc:240] Negotiation successful
0620 23:36:23.652905 (+ 982us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42292,"mutex_wait_us":762,"spinlock_wait_cycles":28672,"thread_start_us":44097,"threads_started":1}
I20260620 23:36:23.656284 7531 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33681
I20260620 23:36:23.656450 7530 heartbeater.cc:347] Connected to a master server at 127.0.97.60:33735
I20260620 23:36:23.656497 7532 heartbeater.cc:347] Connected to a master server at 127.0.97.61:39051
I20260620 23:36:23.656692 7530 heartbeater.cc:464] Registering TS with master...
I20260620 23:36:23.656716 7532 heartbeater.cc:464] Registering TS with master...
I20260620 23:36:23.656692 7531 heartbeater.cc:464] Registering TS with master...
I20260620 23:36:23.657965 7532 heartbeater.cc:511] Master 127.0.97.61:39051 requested a full tablet report, sending...
I20260620 23:36:23.658006 7530 heartbeater.cc:511] Master 127.0.97.60:33735 requested a full tablet report, sending...
I20260620 23:36:23.658030 7531 heartbeater.cc:511] Master 127.0.97.62:33681 requested a full tablet report, sending...
I20260620 23:36:23.660640 7351 ts_manager.cc:194] Registered new tserver with Master: c6c5446de0824443bba78a92059589a9 (127.0.97.1:45847)
I20260620 23:36:23.660656 7160 ts_manager.cc:194] Registered new tserver with Master: c6c5446de0824443bba78a92059589a9 (127.0.97.1:45847)
I20260620 23:36:23.661839 6559 ts_manager.cc:194] Re-registered known tserver with Master: c6c5446de0824443bba78a92059589a9 (127.0.97.1:45847)
I20260620 23:36:23.662839 6559 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:42685
I20260620 23:36:23.744742 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 6754
I20260620 23:36:23.763650 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:42859
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:33681
--webserver_port=42549
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:33681,127.0.97.61:39051
--tserver_master_addrs=127.0.97.62:33681,127.0.97.61:39051,127.0.97.60:33735 with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:36:23.885135 7311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d4ee78c582c4c9c9ae36386fc5afcaf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33681 } } peers { permanent_uuid: "84fa2fff77be49b098f7d05dd4ec1a4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39051 } attrs { promote: false } } peers { permanent_uuid: "4b672bfe633a443591fc2dd3ca7277b1" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 33735 } attrs { promote: true } } }
I20260620 23:36:23.886008 7311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d4ee78c582c4c9c9ae36386fc5afcaf [sys.catalog]: This master's current role is: LEADER
I20260620 23:36:23.887368 7311 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 9d4ee78c582c4c9c9ae36386fc5afcaf: attempting to promote NON_VOTER 4b672bfe633a443591fc2dd3ca7277b1 to VOTER
I20260620 23:36:23.888130 7311 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d4ee78c582c4c9c9ae36386fc5afcaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33681 } } peers { permanent_uuid: "84fa2fff77be49b098f7d05dd4ec1a4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39051 } attrs { promote: false } } peers { permanent_uuid: "4b672bfe633a443591fc2dd3ca7277b1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33735 } attrs { promote: false } }
I20260620 23:36:23.890599 7170 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 84fa2fff77be49b098f7d05dd4ec1a4a [term 1 FOLLOWER]: Refusing update from remote peer 9d4ee78c582c4c9c9ae36386fc5afcaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260620 23:36:23.890661 7361 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4b672bfe633a443591fc2dd3ca7277b1 [term 1 LEARNER]: Refusing update from remote peer 9d4ee78c582c4c9c9ae36386fc5afcaf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260620 23:36:23.891273 7543 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9d4ee78c582c4c9c9ae36386fc5afcaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "84fa2fff77be49b098f7d05dd4ec1a4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39051 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 23:36:23.891722 7544 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9d4ee78c582c4c9c9ae36386fc5afcaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b672bfe633a443591fc2dd3ca7277b1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 23:36:23.895869 7543 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9d4ee78c582c4c9c9ae36386fc5afcaf [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4b672bfe633a443591fc2dd3ca7277b1 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33681 } } peers { permanent_uuid: "84fa2fff77be49b098f7d05dd4ec1a4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39051 } attrs { promote: false } } peers { permanent_uuid: "4b672bfe633a443591fc2dd3ca7277b1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33735 } attrs { promote: false } } }
I20260620 23:36:23.897238 7170 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 84fa2fff77be49b098f7d05dd4ec1a4a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4b672bfe633a443591fc2dd3ca7277b1 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33681 } } peers { permanent_uuid: "84fa2fff77be49b098f7d05dd4ec1a4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39051 } attrs { promote: false } } peers { permanent_uuid: "4b672bfe633a443591fc2dd3ca7277b1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33735 } attrs { promote: false } } }
I20260620 23:36:23.900321 7361 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4b672bfe633a443591fc2dd3ca7277b1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4b672bfe633a443591fc2dd3ca7277b1 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33681 } } peers { permanent_uuid: "84fa2fff77be49b098f7d05dd4ec1a4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39051 } attrs { promote: false } } peers { permanent_uuid: "4b672bfe633a443591fc2dd3ca7277b1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33735 } attrs { promote: false } } }
I20260620 23:36:23.901552 7549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84fa2fff77be49b098f7d05dd4ec1a4a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33681 } } peers { permanent_uuid: "84fa2fff77be49b098f7d05dd4ec1a4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39051 } attrs { promote: false } } peers { permanent_uuid: "4b672bfe633a443591fc2dd3ca7277b1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33735 } attrs { promote: false } } }
I20260620 23:36:23.901953 7549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84fa2fff77be49b098f7d05dd4ec1a4a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:36:23.903277 7544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d4ee78c582c4c9c9ae36386fc5afcaf [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33681 } } peers { permanent_uuid: "84fa2fff77be49b098f7d05dd4ec1a4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39051 } attrs { promote: false } } peers { permanent_uuid: "4b672bfe633a443591fc2dd3ca7277b1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33735 } attrs { promote: false } } }
I20260620 23:36:23.903424 7543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d4ee78c582c4c9c9ae36386fc5afcaf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33681 } } peers { permanent_uuid: "84fa2fff77be49b098f7d05dd4ec1a4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39051 } attrs { promote: false } } peers { permanent_uuid: "4b672bfe633a443591fc2dd3ca7277b1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33735 } attrs { promote: false } } }
I20260620 23:36:23.903589 7544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d4ee78c582c4c9c9ae36386fc5afcaf [sys.catalog]: This master's current role is: LEADER
I20260620 23:36:23.903628 7543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d4ee78c582c4c9c9ae36386fc5afcaf [sys.catalog]: This master's current role is: LEADER
I20260620 23:36:23.903947 7543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d4ee78c582c4c9c9ae36386fc5afcaf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33681 } } peers { permanent_uuid: "84fa2fff77be49b098f7d05dd4ec1a4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39051 } attrs { promote: false } } peers { permanent_uuid: "4b672bfe633a443591fc2dd3ca7277b1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33735 } attrs { promote: false } } }
I20260620 23:36:23.904165 7543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d4ee78c582c4c9c9ae36386fc5afcaf [sys.catalog]: This master's current role is: LEADER
I20260620 23:36:23.907398 7550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b672bfe633a443591fc2dd3ca7277b1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33681 } } peers { permanent_uuid: "84fa2fff77be49b098f7d05dd4ec1a4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39051 } attrs { promote: false } } peers { permanent_uuid: "4b672bfe633a443591fc2dd3ca7277b1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33735 } attrs { promote: false } } }
I20260620 23:36:23.907810 7550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b672bfe633a443591fc2dd3ca7277b1 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 23:36:24.127390 7542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:36:24.127749 7542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:36:24.127923 7542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:36:24.135469 7542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:36:24.135637 7542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:36:24.135807 7542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 23:36:24.146680 7542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:42859
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=42549
--tserver_master_addrs=127.0.97.62:33681,127.0.97.61:39051,127.0.97.60:33735
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:36:24.148190 7542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:36:24.149626 7542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:36:24.164321 7558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:24.164400 7555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:24.164389 7556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:36:24.170403 7542 server_base.cc:1034] running on GCE node
I20260620 23:36:24.171248 7542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:36:24.172323 7542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:36:24.173532 7542 hybrid_clock.cc:648] HybridClock initialized: now 1781998584173479 us; error 42 us; skew 500 ppm
I20260620 23:36:24.173887 7542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:36:24 dist-test-slave-tcgx krb5kdc[6514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998584, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:36:24.177073 7542 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260620 23:36:24.178655 7542 webserver.cc:460] Webserver started at http://127.0.97.2:42549/ using document root <none> and password file <none>
I20260620 23:36:24.179477 7542 fs_manager.cc:359] Metadata directory not provided
I20260620 23:36:24.179597 7542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:36:24.185284 7542 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260620 23:36:24.188232 7565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:24.189687 7542 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 23:36:24.189869 7542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-1/wal
uuid: "4290f95cc39945379be798778eae2e18"
format_stamp: "Formatted at 2026-06-20 23:36:20 on dist-test-slave-tcgx"
server_key: "91f133bc51b205ad1ca2f634e49b3e92"
server_key_iv: "a37887d76534f4bc3a3b20ad5d1894c9"
server_key_version: "encryptionkey@0"
I20260620 23:36:24.190889 7542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:36:24.207134 7542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:36:24.212044 7542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:36:24.212343 7542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:36:24.213408 7542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:36:24.215087 7542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:36:24.215200 7542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:24.215323 7542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:36:24.215420 7542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:24.249648 7542 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42859
I20260620 23:36:24.249747 7677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42859 every 8 connection(s)
I20260620 23:36:24.251212 7542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
I20260620 23:36:24.256198 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 7542
I20260620 23:36:24.256502 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 6888
Jun 20 23:36:24 dist-test-slave-tcgx krb5kdc[6514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998584, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Jun 20 23:36:24 dist-test-slave-tcgx krb5kdc[6514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998584, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Jun 20 23:36:24 dist-test-slave-tcgx krb5kdc[6514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998584, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 23:36:24.298228 7687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:24.255753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47737 (local address 127.0.97.60:33735)
0620 23:36:24.278438 (+ 22685us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:24.278448 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:24.278474 (+ 26us) server_negotiation.cc:407] Connection header received
0620 23:36:24.278542 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:24.278548 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:24.278650 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:24.278788 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:24.279940 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:24.280682 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:24.282302 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:24.282906 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:24.283079 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:24.288521 (+ 5442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:24.288530 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:24.288537 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:24.288579 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:24.293309 (+ 4730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:24.294536 (+ 1227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:24.294542 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:24.294545 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:24.295295 (+ 750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:24.295797 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:24.295802 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:24.295806 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:24.296150 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:24.296277 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:24.297577 (+ 1300us) server_negotiation.cc:299] Negotiation successful
0620 23:36:24.297811 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22527,"thread_start_us":23216,"threads_started":1}
I20260620 23:36:24.298796 7683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:24.254189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56685 (local address 127.0.97.61:39051)
0620 23:36:24.274561 (+ 20372us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:24.274566 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:24.274585 (+ 19us) server_negotiation.cc:407] Connection header received
0620 23:36:24.274635 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:24.274639 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:24.274705 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:24.274849 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:24.276977 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:24.277712 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:24.280886 (+ 3174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:24.281577 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:24.281704 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:24.289357 (+ 7653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:24.289365 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:24.289371 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:24.289420 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:24.294064 (+ 4644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:24.294957 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:24.294963 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:24.294967 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:24.295127 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:24.295807 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:24.295814 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:24.295818 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:24.296287 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:24.296434 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:24.297734 (+ 1300us) server_negotiation.cc:299] Negotiation successful
0620 23:36:24.297978 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20265,"thread_start_us":20103,"threads_started":1}
I20260620 23:36:24.300514 7685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:24.255001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33735 (local address 127.0.97.2:47737)
0620 23:36:24.267267 (+ 12266us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:24.267341 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:24.273682 (+ 6341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:24.278840 (+ 5158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:24.278845 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:24.279369 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:24.279728 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:24.279743 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:24.280828 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:24.280835 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:24.282146 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:24.282158 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:24.283347 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:24.283358 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:24.283621 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:24.284232 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:24.284261 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:24.288237 (+ 3976us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:36:24.293522 (+ 5285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:24.293534 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:24.293556 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:24.294381 (+ 825us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:24.295496 (+ 1115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:24.295503 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:24.295507 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:24.295669 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:24.296358 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:36:24.296367 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:36:24.296927 (+ 560us) client_negotiation.cc:769] Sending connection context
0620 23:36:24.297258 (+ 331us) client_negotiation.cc:240] Negotiation successful
0620 23:36:24.298793 (+ 1535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11861,"mutex_wait_us":1048,"spinlock_wait_cycles":25856,"thread_start_us":23957,"threads_started":1}
I20260620 23:36:24.300648 7686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:24.255230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39051 (local address 127.0.97.2:56685)
0620 23:36:24.270145 (+ 14915us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:24.270165 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:24.273682 (+ 3517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:24.275218 (+ 1536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:24.275229 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:24.276070 (+ 841us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:24.276742 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:24.276761 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:24.277897 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:24.277901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:24.279842 (+ 1941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:24.279852 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:24.282422 (+ 2570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:24.282428 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:24.282727 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:24.283666 (+ 939us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:24.283712 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:24.289107 (+ 5395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:36:24.294240 (+ 5133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:24.294245 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:24.294248 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:24.294758 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:24.295279 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:24.295283 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:24.295335 (+ 52us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:24.295653 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:24.296496 (+ 843us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:36:24.296501 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:36:24.296921 (+ 420us) client_negotiation.cc:769] Sending connection context
0620 23:36:24.297316 (+ 395us) client_negotiation.cc:240] Negotiation successful
0620 23:36:24.299758 (+ 2442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14787,"mutex_wait_us":11,"spinlock_wait_cycles":66432,"thread_start_us":11510,"threads_started":1}
I20260620 23:36:24.302234 7684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:24.255002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33681 (local address 127.0.97.2:45455)
0620 23:36:24.279499 (+ 24497us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:24.279528 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:24.279679 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:24.283158 (+ 3479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:24.283165 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:24.283960 (+ 795us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:24.285636 (+ 1676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:24.285649 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:24.287891 (+ 2242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:24.287896 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:24.289605 (+ 1709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:24.289623 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:24.290716 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:24.290724 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:24.290975 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:24.291597 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:24.291625 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:24.294367 (+ 2742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:36:24.298900 (+ 4533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:24.298907 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:24.298913 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:24.299413 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:24.300161 (+ 748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:24.300168 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:24.300175 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:24.300365 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:24.301049 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:36:24.301054 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:36:24.301222 (+ 168us) client_negotiation.cc:769] Sending connection context
0620 23:36:24.301369 (+ 147us) client_negotiation.cc:240] Negotiation successful
0620 23:36:24.301842 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23698,"thread_start_us":12782,"threads_started":1}
I20260620 23:36:24.302304 7678 heartbeater.cc:347] Connected to a master server at 127.0.97.60:33735
I20260620 23:36:24.302651 7678 heartbeater.cc:464] Registering TS with master...
I20260620 23:36:24.302919 7682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:24.254138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45455 (local address 127.0.97.62:33681)
0620 23:36:24.282556 (+ 28418us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:24.282565 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:24.282656 (+ 91us) server_negotiation.cc:407] Connection header received
0620 23:36:24.282723 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:24.282732 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:24.282839 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:24.284884 (+ 2045us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:24.285811 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:24.287258 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:24.289804 (+ 2546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:24.290458 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:24.290646 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:24.294999 (+ 4353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:24.295007 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:24.295014 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:24.295067 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:24.298707 (+ 3640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:24.299559 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:24.299565 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:24.299568 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:24.299729 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:24.300556 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:24.300561 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:24.300564 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:24.300892 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:24.301060 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:24.302323 (+ 1263us) server_negotiation.cc:299] Negotiation successful
0620 23:36:24.302597 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28260,"thread_start_us":28813,"threads_started":1}
I20260620 23:36:24.303260 7680 heartbeater.cc:347] Connected to a master server at 127.0.97.61:39051
I20260620 23:36:24.303397 7680 heartbeater.cc:464] Registering TS with master...
I20260620 23:36:24.303750 7678 heartbeater.cc:511] Master 127.0.97.60:33735 requested a full tablet report, sending...
I20260620 23:36:24.303802 7679 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33681
I20260620 23:36:24.303923 7679 heartbeater.cc:464] Registering TS with master...
I20260620 23:36:24.304335 7679 heartbeater.cc:511] Master 127.0.97.62:33681 requested a full tablet report, sending...
Jun 20 23:36:24 dist-test-slave-tcgx krb5kdc[6514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260620 23:36:24.305781 7680 heartbeater.cc:511] Master 127.0.97.61:39051 requested a full tablet report, sending...
I20260620 23:36:24.305776 7351 ts_manager.cc:194] Registered new tserver with Master: 4290f95cc39945379be798778eae2e18 (127.0.97.2:42859)
I20260620 23:36:24.307137 7160 ts_manager.cc:194] Registered new tserver with Master: 4290f95cc39945379be798778eae2e18 (127.0.97.2:42859)
I20260620 23:36:24.308656 6559 ts_manager.cc:194] Re-registered known tserver with Master: 4290f95cc39945379be798778eae2e18 (127.0.97.2:42859)
I20260620 23:36:24.309769 6559 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:45455
I20260620 23:36:24.313977 7689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:24.258097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58370 (local address 127.0.97.3:36767)
0620 23:36:24.285523 (+ 27426us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:24.285535 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:24.290365 (+ 4830us) server_negotiation.cc:407] Connection header received
0620 23:36:24.291295 (+ 930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:24.291309 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:24.291697 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:24.291878 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:24.294231 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:24.296348 (+ 2117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:24.298206 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:24.299154 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:24.299325 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:24.306513 (+ 7188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:24.306536 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:24.306553 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:24.306603 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:24.310630 (+ 4027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:24.311590 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:24.311600 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:24.311604 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:24.311769 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:24.312283 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:24.312287 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:24.312291 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:24.312678 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:24.312895 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:24.313353 (+ 458us) server_negotiation.cc:299] Negotiation successful
0620 23:36:24.313579 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27211,"thread_start_us":27426,"threads_started":1}
I20260620 23:36:24.315565 7011 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:36:24.401230 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 6888
I20260620 23:36:24.418286 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:36767
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:33681
--webserver_port=44257
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:33681,127.0.97.61:39051
--tserver_master_addrs=127.0.97.62:33681,127.0.97.61:39051,127.0.97.60:33735 with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:36:24.665273 7531 heartbeater.cc:503] Master 127.0.97.62:33681 was elected leader, sending a full tablet report...
W20260620 23:36:24.755929 7691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:36:24.756228 7691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:36:24.756321 7691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:36:24.764135 7691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:36:24.764287 7691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:36:24.764462 7691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 23:36:24.774835 7691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:36767
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=44257
--tserver_master_addrs=127.0.97.62:33681,127.0.97.61:39051,127.0.97.60:33735
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:36:24.776314 7691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:36:24.777647 7691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:36:24.791293 7696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:24.791368 7697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:24.791324 7699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:36:24.792567 7691 server_base.cc:1034] running on GCE node
I20260620 23:36:24.793195 7691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:36:24.794214 7691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:36:24.795454 7691 hybrid_clock.cc:648] HybridClock initialized: now 1781998584795409 us; error 56 us; skew 500 ppm
I20260620 23:36:24.795835 7691 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:36:24 dist-test-slave-tcgx krb5kdc[6514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998584, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:36:24.799163 7691 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260620 23:36:24.800889 7691 webserver.cc:460] Webserver started at http://127.0.97.3:44257/ using document root <none> and password file <none>
I20260620 23:36:24.801661 7691 fs_manager.cc:359] Metadata directory not provided
I20260620 23:36:24.801774 7691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:36:24.807381 7691 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:36:24.810403 7706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:24.811905 7691 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:36:24.812044 7691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-2/wal
uuid: "7fa1b50d6220465ab5e6409556e6ee7e"
format_stamp: "Formatted at 2026-06-20 23:36:21 on dist-test-slave-tcgx"
server_key: "49829f1c7fb45416d851bacd0b1116ca"
server_key_iv: "29a3960ccffbd149120a7789c2189ae3"
server_key_version: "encryptionkey@0"
I20260620 23:36:24.813028 7691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:36:24.833352 7691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:36:24.837711 7691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:36:24.838001 7691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:36:24.839049 7691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:36:24.840539 7691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:36:24.840667 7691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:24.840794 7691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:36:24.840878 7691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:24.872845 7691 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36767
I20260620 23:36:24.872938 7818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36767 every 8 connection(s)
I20260620 23:36:24.874331 7691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
I20260620 23:36:24.880187 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 7691
Jun 20 23:36:24 dist-test-slave-tcgx krb5kdc[6514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998584, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Jun 20 23:36:24 dist-test-slave-tcgx krb5kdc[6514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998584, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Jun 20 23:36:24 dist-test-slave-tcgx krb5kdc[6514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998584, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260620 23:36:24.943251 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:24.878183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44351 (local address 127.0.97.61:39051)
0620 23:36:24.907543 (+ 29360us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:24.907553 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:24.907578 (+ 25us) server_negotiation.cc:407] Connection header received
0620 23:36:24.907652 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:24.907660 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:24.907758 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:24.907902 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:24.911068 (+ 3166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:24.912219 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:24.916583 (+ 4364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:24.917520 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:24.917650 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:24.926870 (+ 9220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:24.926878 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:24.926882 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:24.926922 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:24.935591 (+ 8669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:24.939537 (+ 3946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:24.939545 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:24.939553 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:24.939742 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:24.941218 (+ 1476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:24.941226 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:24.941229 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:24.941607 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:24.941740 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:24.942536 (+ 796us) server_negotiation.cc:299] Negotiation successful
0620 23:36:24.942830 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29202,"thread_start_us":29088,"threads_started":1}
I20260620 23:36:24.943444 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:24.877411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40437 (local address 127.0.97.60:33735)
0620 23:36:24.906271 (+ 28860us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:24.906279 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:24.906916 (+ 637us) server_negotiation.cc:407] Connection header received
0620 23:36:24.907172 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:24.907181 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:24.907285 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:24.907408 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:24.911055 (+ 3647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:24.913677 (+ 2622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:24.921092 (+ 7415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:24.922072 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:24.922212 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:24.934109 (+ 11897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:24.934115 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:24.934120 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:24.934170 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:24.937797 (+ 3627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:24.938761 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:24.938766 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:24.938769 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:24.938906 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:24.940873 (+ 1967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:24.940878 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:24.940882 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:24.941311 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:24.941452 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:24.942818 (+ 1366us) server_negotiation.cc:299] Negotiation successful
0620 23:36:24.943081 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28694,"thread_start_us":28596,"threads_started":1}
I20260620 23:36:24.944006 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:24.877324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60777 (local address 127.0.97.62:33681)
0620 23:36:24.908400 (+ 31076us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:24.908405 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:24.908421 (+ 16us) server_negotiation.cc:407] Connection header received
0620 23:36:24.908472 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:24.908475 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:24.908532 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:24.908662 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:24.911255 (+ 2593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:24.912208 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:24.916805 (+ 4597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:24.917273 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:24.917415 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:24.926684 (+ 9269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:24.926695 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:24.926700 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:24.926746 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:24.938459 (+ 11713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:24.939478 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:24.939486 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:24.939493 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:24.939651 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:24.941757 (+ 2106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:24.941765 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:24.941771 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:24.942057 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:24.942213 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:24.942905 (+ 692us) server_negotiation.cc:299] Negotiation successful
0620 23:36:24.943148 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30966,"thread_start_us":30865,"threads_started":1}
I20260620 23:36:24.944706 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:24.878027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33681 (local address 127.0.97.3:60777)
0620 23:36:24.899448 (+ 21421us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:24.899535 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:24.903191 (+ 3656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:24.908689 (+ 5498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:24.908694 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:24.909485 (+ 791us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:24.911108 (+ 1623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:24.911125 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:24.913774 (+ 2649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:24.913780 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:24.916647 (+ 2867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:24.916666 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:24.917546 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:24.917553 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:24.917974 (+ 421us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:24.919252 (+ 1278us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:24.919285 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:24.926327 (+ 7042us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:36:24.938824 (+ 12497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:24.938829 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:24.938832 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:24.939378 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:24.940471 (+ 1093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:24.940476 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:24.940480 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:24.941038 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:24.942177 (+ 1139us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:36:24.942181 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:36:24.942359 (+ 178us) client_negotiation.cc:769] Sending connection context
0620 23:36:24.942490 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 23:36:24.943100 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20962,"mutex_wait_us":338,"thread_start_us":27517,"threads_started":1}
I20260620 23:36:24.944731 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:24.878033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33735 (local address 127.0.97.3:40437)
0620 23:36:24.906751 (+ 28718us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:24.906778 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:24.906993 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:24.908887 (+ 1894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:24.908894 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:24.909572 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:24.910802 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:24.910817 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:24.913859 (+ 3042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:24.913865 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:24.918869 (+ 5004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:24.918890 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:24.923291 (+ 4401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:24.923300 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:24.923561 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:24.924901 (+ 1340us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:24.924934 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:24.933873 (+ 8939us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:36:24.937943 (+ 4070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:24.937949 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:24.937966 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:24.938567 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:24.939087 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:24.939092 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:24.939096 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:24.940737 (+ 1641us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:24.941542 (+ 805us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:36:24.941552 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:36:24.942089 (+ 537us) client_negotiation.cc:769] Sending connection context
0620 23:36:24.942415 (+ 326us) client_negotiation.cc:240] Negotiation successful
0620 23:36:24.943933 (+ 1518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28524,"mutex_wait_us":4259,"spinlock_wait_cycles":24960,"thread_start_us":25373,"threads_started":1}
I20260620 23:36:24.945686 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:24.878321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39051 (local address 127.0.97.3:44351)
0620 23:36:24.905114 (+ 26793us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:24.905144 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:24.905292 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:24.908126 (+ 2834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:24.908148 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:24.909570 (+ 1422us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:24.910768 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:24.910800 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:24.912922 (+ 2122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:24.912931 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:24.916351 (+ 3420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:24.916374 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:24.918066 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:24.918073 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:24.918254 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:24.919105 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:24.919150 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:24.926641 (+ 7491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:36:24.935865 (+ 9224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:24.935881 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:24.935912 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:24.936869 (+ 957us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:24.939972 (+ 3103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:24.939979 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:24.939984 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:24.940359 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:24.941748 (+ 1389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:36:24.941752 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:36:24.942080 (+ 328us) client_negotiation.cc:769] Sending connection context
0620 23:36:24.943527 (+ 1447us) client_negotiation.cc:240] Negotiation successful
0620 23:36:24.944797 (+ 1270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26621,"mutex_wait_us":1056,"spinlock_wait_cycles":27904,"thread_start_us":28013,"threads_started":1}
I20260620 23:36:24.946605 7819 heartbeater.cc:347] Connected to a master server at 127.0.97.60:33735
I20260620 23:36:24.946625 7820 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33681
I20260620 23:36:24.946977 7820 heartbeater.cc:464] Registering TS with master...
I20260620 23:36:24.947013 7819 heartbeater.cc:464] Registering TS with master...
I20260620 23:36:24.947167 7821 heartbeater.cc:347] Connected to a master server at 127.0.97.61:39051
I20260620 23:36:24.947304 7821 heartbeater.cc:464] Registering TS with master...
I20260620 23:36:24.948098 7821 heartbeater.cc:511] Master 127.0.97.61:39051 requested a full tablet report, sending...
I20260620 23:36:24.948140 7819 heartbeater.cc:511] Master 127.0.97.60:33735 requested a full tablet report, sending...
I20260620 23:36:24.948171 7820 heartbeater.cc:511] Master 127.0.97.62:33681 requested a full tablet report, sending...
I20260620 23:36:24.950088 7160 ts_manager.cc:194] Registered new tserver with Master: 7fa1b50d6220465ab5e6409556e6ee7e (127.0.97.3:36767)
I20260620 23:36:24.950389 6559 ts_manager.cc:194] Re-registered known tserver with Master: 7fa1b50d6220465ab5e6409556e6ee7e (127.0.97.3:36767)
I20260620 23:36:24.951516 6559 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:60777
I20260620 23:36:24.952051 7351 ts_manager.cc:194] Registered new tserver with Master: 7fa1b50d6220465ab5e6409556e6ee7e (127.0.97.3:36767)
I20260620 23:36:24.959540 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:24.919416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34168 (local address 127.0.97.60:33735)
0620 23:36:24.928317 (+ 8901us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:24.928326 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:24.929219 (+ 893us) server_negotiation.cc:407] Connection header received
0620 23:36:24.929341 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:24.929351 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:24.929458 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:24.929582 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:24.934101 (+ 4519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:24.935488 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:24.937741 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:24.938401 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:24.938542 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:24.952267 (+ 13725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:24.952274 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:24.952279 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:24.952319 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:24.956441 (+ 4122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:24.957362 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:24.957368 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:24.957372 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:24.957509 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:24.957877 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:24.957882 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:24.957886 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:24.958147 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:24.958390 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:24.958890 (+ 500us) server_negotiation.cc:299] Negotiation successful
0620 23:36:24.959143 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8723,"thread_start_us":8605,"threads_started":1}
I20260620 23:36:24.959635 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:24.912221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56470 (local address 127.0.97.61:39051)
0620 23:36:24.930186 (+ 17965us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:24.930193 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:24.930213 (+ 20us) server_negotiation.cc:407] Connection header received
0620 23:36:24.930274 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:24.930281 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:24.930359 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:24.931542 (+ 1183us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:24.933334 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:24.940099 (+ 6765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:24.945172 (+ 5073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:24.945867 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:24.945985 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:24.952227 (+ 6242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:24.952234 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:24.952239 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:24.952279 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:24.956066 (+ 3787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:24.956797 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:24.956807 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:24.956810 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:24.956952 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:24.957392 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:24.957399 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:24.957403 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:24.957761 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:24.957995 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:24.958587 (+ 592us) server_negotiation.cc:299] Negotiation successful
0620 23:36:24.958822 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17839,"thread_start_us":17697,"threads_started":1}
I20260620 23:36:24.963874 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:24.912544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39400 (local address 127.0.97.62:33681)
0620 23:36:24.938901 (+ 26357us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:24.938909 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:24.938928 (+ 19us) server_negotiation.cc:407] Connection header received
0620 23:36:24.939014 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:24.939020 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:24.939148 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:24.939261 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:24.948450 (+ 9189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:24.949546 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:24.951533 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:24.953887 (+ 2354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:24.954027 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:24.957422 (+ 3395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:24.957429 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:24.957435 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:24.957480 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:24.961082 (+ 3602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:24.961922 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:24.961926 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:24.961929 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:24.962047 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:24.962414 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:24.962420 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:24.962424 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:24.962812 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:24.963086 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:24.963359 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 23:36:24.963542 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26235,"thread_start_us":26080,"threads_started":1}
I20260620 23:36:24.968487 6559 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39400:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:36:24.975477 6559 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:36:25.015985 7850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:24.996934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42859 (local address 127.0.0.1:55884)
0620 23:36:25.000048 (+ 3114us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:25.000074 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:25.000251 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:25.006488 (+ 6237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:25.006497 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:25.006539 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:36:25.007065 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:25.007084 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:25.010450 (+ 3366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:25.010456 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:25.012747 (+ 2291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:25.012762 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:25.014713 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:25.014719 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:25.014889 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:25.014922 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 23:36:25.015032 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 23:36:25.015183 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":349,"threads_started":1}
I20260620 23:36:25.015992 7851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:24.997141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55884 (local address 127.0.97.2:42859)
0620 23:36:25.005581 (+ 8440us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:25.005595 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:25.005634 (+ 39us) server_negotiation.cc:407] Connection header received
0620 23:36:25.005778 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:25.005791 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:25.006169 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:25.007391 (+ 1222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:36:25.007681 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:25.010299 (+ 2618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:25.012923 (+ 2624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:25.014498 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:25.014663 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:25.014876 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:25.015102 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 23:36:25.015278 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8213,"thread_start_us":8060,"threads_started":1}
I20260620 23:36:25.017309 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:24.997200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45847 (local address 127.0.0.1:55320)
0620 23:36:25.000992 (+ 3792us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:25.001258 (+ 266us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:25.001710 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:25.008363 (+ 6653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:25.008368 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:25.008395 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:36:25.008759 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:25.008772 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:25.010769 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:25.010775 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:25.013044 (+ 2269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:25.013058 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:25.016026 (+ 2968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:25.016045 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:25.016672 (+ 627us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:25.016704 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 23:36:25.016791 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 23:36:25.016926 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3366,"mutex_wait_us":1,"thread_start_us":3215,"threads_started":1}
I20260620 23:36:25.018074 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:24.997732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55320 (local address 127.0.97.1:45847)
0620 23:36:25.001922 (+ 4190us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:25.001933 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:25.002010 (+ 77us) server_negotiation.cc:407] Connection header received
0620 23:36:25.002157 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:25.002171 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:25.008165 (+ 5994us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:25.008383 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:36:25.009050 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:25.010618 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:25.013221 (+ 2603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:25.015011 (+ 1790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:25.015139 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:25.015332 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:25.017605 (+ 2273us) server_negotiation.cc:299] Negotiation successful
0620 23:36:25.017777 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3625,"thread_start_us":3769,"threads_started":1}
I20260620 23:36:25.018612 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:25.000833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36767 (local address 127.0.0.1:58374)
0620 23:36:25.007494 (+ 6661us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:25.007516 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:25.007648 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:25.011052 (+ 3404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:25.011056 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:25.011084 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:36:25.011488 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:25.011498 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:25.012976 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:25.012981 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:25.016113 (+ 3132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:25.016128 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:25.017990 (+ 1862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:25.017995 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:25.018113 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:25.018169 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 23:36:25.018253 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 23:36:25.018377 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6518,"thread_start_us":6359,"threads_started":1}
I20260620 23:36:25.019384 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:25.001199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58374 (local address 127.0.97.3:36767)
0620 23:36:25.010209 (+ 9010us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:25.010219 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:25.010252 (+ 33us) server_negotiation.cc:407] Connection header received
0620 23:36:25.010346 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:25.010354 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:25.010624 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:25.010820 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:36:25.011700 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:25.012854 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:25.016300 (+ 3446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:25.017489 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:25.017609 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:25.017862 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:25.018874 (+ 1012us) server_negotiation.cc:299] Negotiation successful
0620 23:36:25.019060 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8809,"thread_start_us":8613,"threads_started":1}
I20260620 23:36:25.020299 7613 tablet_service.cc:1467] Processing CreateTablet for tablet bc037ed734fd4843899753f7dbc8f863 (DEFAULT_TABLE table=test-table [id=9c24694f672546c29a12f2ece3b0fffb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:36:25.021788 7613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc037ed734fd4843899753f7dbc8f863. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:36:25.021746 7465 tablet_service.cc:1467] Processing CreateTablet for tablet bc037ed734fd4843899753f7dbc8f863 (DEFAULT_TABLE table=test-table [id=9c24694f672546c29a12f2ece3b0fffb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:36:25.023188 7465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc037ed734fd4843899753f7dbc8f863. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:36:25.024000 7754 tablet_service.cc:1467] Processing CreateTablet for tablet bc037ed734fd4843899753f7dbc8f863 (DEFAULT_TABLE table=test-table [id=9c24694f672546c29a12f2ece3b0fffb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:36:25.025393 7754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc037ed734fd4843899753f7dbc8f863. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:36:25.033648 7856 tablet_bootstrap.cc:492] T bc037ed734fd4843899753f7dbc8f863 P 4290f95cc39945379be798778eae2e18: Bootstrap starting.
I20260620 23:36:25.037045 7856 tablet_bootstrap.cc:654] T bc037ed734fd4843899753f7dbc8f863 P 4290f95cc39945379be798778eae2e18: Neither blocks nor log segments found. Creating new log.
I20260620 23:36:25.038790 7856 log.cc:824] T bc037ed734fd4843899753f7dbc8f863 P 4290f95cc39945379be798778eae2e18: Log is configured to *not* fsync() on all Append() calls
I20260620 23:36:25.041916 7856 tablet_bootstrap.cc:492] T bc037ed734fd4843899753f7dbc8f863 P 4290f95cc39945379be798778eae2e18: No bootstrap required, opened a new log
I20260620 23:36:25.042197 7856 ts_tablet_manager.cc:1397] T bc037ed734fd4843899753f7dbc8f863 P 4290f95cc39945379be798778eae2e18: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 23:36:25.043073 7859 tablet_bootstrap.cc:492] T bc037ed734fd4843899753f7dbc8f863 P 7fa1b50d6220465ab5e6409556e6ee7e: Bootstrap starting.
I20260620 23:36:25.043110 7857 tablet_bootstrap.cc:492] T bc037ed734fd4843899753f7dbc8f863 P c6c5446de0824443bba78a92059589a9: Bootstrap starting.
I20260620 23:36:25.045986 7857 tablet_bootstrap.cc:654] T bc037ed734fd4843899753f7dbc8f863 P c6c5446de0824443bba78a92059589a9: Neither blocks nor log segments found. Creating new log.
I20260620 23:36:25.046727 7859 tablet_bootstrap.cc:654] T bc037ed734fd4843899753f7dbc8f863 P 7fa1b50d6220465ab5e6409556e6ee7e: Neither blocks nor log segments found. Creating new log.
I20260620 23:36:25.047719 7857 log.cc:824] T bc037ed734fd4843899753f7dbc8f863 P c6c5446de0824443bba78a92059589a9: Log is configured to *not* fsync() on all Append() calls
I20260620 23:36:25.048043 7859 log.cc:824] T bc037ed734fd4843899753f7dbc8f863 P 7fa1b50d6220465ab5e6409556e6ee7e: Log is configured to *not* fsync() on all Append() calls
I20260620 23:36:25.052697 7857 tablet_bootstrap.cc:492] T bc037ed734fd4843899753f7dbc8f863 P c6c5446de0824443bba78a92059589a9: No bootstrap required, opened a new log
I20260620 23:36:25.053040 7857 ts_tablet_manager.cc:1397] T bc037ed734fd4843899753f7dbc8f863 P c6c5446de0824443bba78a92059589a9: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20260620 23:36:25.054396 7856 raft_consensus.cc:348] T bc037ed734fd4843899753f7dbc8f863 P 4290f95cc39945379be798778eae2e18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6c5446de0824443bba78a92059589a9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45847 } } peers { permanent_uuid: "4290f95cc39945379be798778eae2e18" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42859 } } peers { permanent_uuid: "7fa1b50d6220465ab5e6409556e6ee7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36767 } }
I20260620 23:36:25.054875 7856 raft_consensus.cc:374] T bc037ed734fd4843899753f7dbc8f863 P 4290f95cc39945379be798778eae2e18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:36:25.055008 7856 raft_consensus.cc:729] T bc037ed734fd4843899753f7dbc8f863 P 4290f95cc39945379be798778eae2e18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4290f95cc39945379be798778eae2e18, State: Initialized, Role: FOLLOWER
I20260620 23:36:25.055583 7859 tablet_bootstrap.cc:492] T bc037ed734fd4843899753f7dbc8f863 P 7fa1b50d6220465ab5e6409556e6ee7e: No bootstrap required, opened a new log
I20260620 23:36:25.055675 7856 consensus_queue.cc:260] T bc037ed734fd4843899753f7dbc8f863 P 4290f95cc39945379be798778eae2e18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6c5446de0824443bba78a92059589a9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45847 } } peers { permanent_uuid: "4290f95cc39945379be798778eae2e18" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42859 } } peers { permanent_uuid: "7fa1b50d6220465ab5e6409556e6ee7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36767 } }
I20260620 23:36:25.055861 7859 ts_tablet_manager.cc:1397] T bc037ed734fd4843899753f7dbc8f863 P 7fa1b50d6220465ab5e6409556e6ee7e: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.003s
I20260620 23:36:25.057668 7856 ts_tablet_manager.cc:1428] T bc037ed734fd4843899753f7dbc8f863 P 4290f95cc39945379be798778eae2e18: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260620 23:36:25.057951 7679 heartbeater.cc:503] Master 127.0.97.62:33681 was elected leader, sending a full tablet report...
I20260620 23:36:25.063361 7857 raft_consensus.cc:348] T bc037ed734fd4843899753f7dbc8f863 P c6c5446de0824443bba78a92059589a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6c5446de0824443bba78a92059589a9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45847 } } peers { permanent_uuid: "4290f95cc39945379be798778eae2e18" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42859 } } peers { permanent_uuid: "7fa1b50d6220465ab5e6409556e6ee7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36767 } }
I20260620 23:36:25.063745 7857 raft_consensus.cc:374] T bc037ed734fd4843899753f7dbc8f863 P c6c5446de0824443bba78a92059589a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:36:25.063845 7857 raft_consensus.cc:729] T bc037ed734fd4843899753f7dbc8f863 P c6c5446de0824443bba78a92059589a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6c5446de0824443bba78a92059589a9, State: Initialized, Role: FOLLOWER
I20260620 23:36:25.064479 7857 consensus_queue.cc:260] T bc037ed734fd4843899753f7dbc8f863 P c6c5446de0824443bba78a92059589a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6c5446de0824443bba78a92059589a9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45847 } } peers { permanent_uuid: "4290f95cc39945379be798778eae2e18" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42859 } } peers { permanent_uuid: "7fa1b50d6220465ab5e6409556e6ee7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36767 } }
I20260620 23:36:25.066397 7857 ts_tablet_manager.cc:1428] T bc037ed734fd4843899753f7dbc8f863 P c6c5446de0824443bba78a92059589a9: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260620 23:36:25.072448 7859 raft_consensus.cc:348] T bc037ed734fd4843899753f7dbc8f863 P 7fa1b50d6220465ab5e6409556e6ee7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6c5446de0824443bba78a92059589a9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45847 } } peers { permanent_uuid: "4290f95cc39945379be798778eae2e18" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42859 } } peers { permanent_uuid: "7fa1b50d6220465ab5e6409556e6ee7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36767 } }
I20260620 23:36:25.072855 7859 raft_consensus.cc:374] T bc037ed734fd4843899753f7dbc8f863 P 7fa1b50d6220465ab5e6409556e6ee7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:36:25.072974 7859 raft_consensus.cc:729] T bc037ed734fd4843899753f7dbc8f863 P 7fa1b50d6220465ab5e6409556e6ee7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa1b50d6220465ab5e6409556e6ee7e, State: Initialized, Role: FOLLOWER
I20260620 23:36:25.073566 7859 consensus_queue.cc:260] T bc037ed734fd4843899753f7dbc8f863 P 7fa1b50d6220465ab5e6409556e6ee7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6c5446de0824443bba78a92059589a9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45847 } } peers { permanent_uuid: "4290f95cc39945379be798778eae2e18" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42859 } } peers { permanent_uuid: "7fa1b50d6220465ab5e6409556e6ee7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36767 } }
I20260620 23:36:25.074677 7820 heartbeater.cc:503] Master 127.0.97.62:33681 was elected leader, sending a full tablet report...
I20260620 23:36:25.075312 7859 ts_tablet_manager.cc:1428] T bc037ed734fd4843899753f7dbc8f863 P 7fa1b50d6220465ab5e6409556e6ee7e: Time spent starting tablet: real 0.019s user 0.005s sys 0.009s
W20260620 23:36:25.115410 7533 tablet.cc:2369] T bc037ed734fd4843899753f7dbc8f863 P c6c5446de0824443bba78a92059589a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:36:25.127696 7822 tablet.cc:2369] T bc037ed734fd4843899753f7dbc8f863 P 7fa1b50d6220465ab5e6409556e6ee7e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:36:25.143481 7863 raft_consensus.cc:482] T bc037ed734fd4843899753f7dbc8f863 P c6c5446de0824443bba78a92059589a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:36:25.143782 7863 raft_consensus.cc:504] T bc037ed734fd4843899753f7dbc8f863 P c6c5446de0824443bba78a92059589a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6c5446de0824443bba78a92059589a9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45847 } } peers { permanent_uuid: "4290f95cc39945379be798778eae2e18" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42859 } } peers { permanent_uuid: "7fa1b50d6220465ab5e6409556e6ee7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36767 } }
I20260620 23:36:25.145960 7863 leader_election.cc:288] T bc037ed734fd4843899753f7dbc8f863 P c6c5446de0824443bba78a92059589a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4290f95cc39945379be798778eae2e18 (127.0.97.2:42859), 7fa1b50d6220465ab5e6409556e6ee7e (127.0.97.3:36767)
I20260620 23:36:25.152849 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:25.146386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40541 (local address 127.0.97.3:36767)
0620 23:36:25.146626 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:25.146635 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:25.147312 (+ 677us) server_negotiation.cc:407] Connection header received
0620 23:36:25.147541 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:25.147545 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:25.147611 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:25.147680 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:36:25.148287 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:25.149021 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:25.151132 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:25.151879 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:25.152034 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:25.152148 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:25.152472 (+ 324us) server_negotiation.cc:299] Negotiation successful
0620 23:36:25.152605 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 23:36:25.152909 7851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:25.146357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36151 (local address 127.0.97.2:42859)
0620 23:36:25.146577 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:25.146584 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:25.147356 (+ 772us) server_negotiation.cc:407] Connection header received
0620 23:36:25.147558 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:25.147562 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:25.147619 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:25.147721 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:36:25.148358 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:25.149039 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:25.151108 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:25.151876 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:25.152041 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:25.152161 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:25.152513 (+ 352us) server_negotiation.cc:299] Negotiation successful
0620 23:36:25.152640 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 23:36:25.153039 7865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:25.146216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36767 (local address 127.0.97.1:40541)
0620 23:36:25.147178 (+ 962us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:25.147195 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:25.147363 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:25.147725 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:25.147729 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:25.147753 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:36:25.148143 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:25.148155 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:25.149168 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:25.149176 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:25.150919 (+ 1743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:25.150983 (+ 64us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:25.152125 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:25.152133 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:25.152307 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:25.152338 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 23:36:25.152447 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 23:36:25.152578 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":827,"mutex_wait_us":101,"thread_start_us":564,"threads_started":1}
I20260620 23:36:25.153079 7866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:25.146218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42859 (local address 127.0.97.1:36151)
0620 23:36:25.147211 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:25.147228 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:25.147394 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:25.147752 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:25.147756 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:25.147778 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:36:25.148178 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:25.148187 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:25.149235 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:25.149239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:25.150914 (+ 1675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:25.150926 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:25.152166 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:25.152174 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:25.152351 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:25.152382 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 23:36:25.152487 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 23:36:25.152620 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":894,"mutex_wait_us":296,"thread_start_us":409,"threads_started":1}
I20260620 23:36:25.154017 7774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc037ed734fd4843899753f7dbc8f863" candidate_uuid: "c6c5446de0824443bba78a92059589a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa1b50d6220465ab5e6409556e6ee7e" is_pre_election: true
I20260620 23:36:25.154019 7633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc037ed734fd4843899753f7dbc8f863" candidate_uuid: "c6c5446de0824443bba78a92059589a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4290f95cc39945379be798778eae2e18" is_pre_election: true
I20260620 23:36:25.154515 7633 raft_consensus.cc:2454] T bc037ed734fd4843899753f7dbc8f863 P 4290f95cc39945379be798778eae2e18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6c5446de0824443bba78a92059589a9 in term 0.
I20260620 23:36:25.154515 7774 raft_consensus.cc:2454] T bc037ed734fd4843899753f7dbc8f863 P 7fa1b50d6220465ab5e6409556e6ee7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6c5446de0824443bba78a92059589a9 in term 0.
I20260620 23:36:25.155246 7421 leader_election.cc:302] T bc037ed734fd4843899753f7dbc8f863 P c6c5446de0824443bba78a92059589a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4290f95cc39945379be798778eae2e18, 7fa1b50d6220465ab5e6409556e6ee7e, c6c5446de0824443bba78a92059589a9; no voters:
I20260620 23:36:25.155762 7863 raft_consensus.cc:2781] T bc037ed734fd4843899753f7dbc8f863 P c6c5446de0824443bba78a92059589a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:36:25.155902 7863 raft_consensus.cc:482] T bc037ed734fd4843899753f7dbc8f863 P c6c5446de0824443bba78a92059589a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:36:25.156018 7863 raft_consensus.cc:3037] T bc037ed734fd4843899753f7dbc8f863 P c6c5446de0824443bba78a92059589a9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:36:25.158187 7863 raft_consensus.cc:504] T bc037ed734fd4843899753f7dbc8f863 P c6c5446de0824443bba78a92059589a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6c5446de0824443bba78a92059589a9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45847 } } peers { permanent_uuid: "4290f95cc39945379be798778eae2e18" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42859 } } peers { permanent_uuid: "7fa1b50d6220465ab5e6409556e6ee7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36767 } }
I20260620 23:36:25.158854 7863 leader_election.cc:288] T bc037ed734fd4843899753f7dbc8f863 P c6c5446de0824443bba78a92059589a9 [CANDIDATE]: Term 1 election: Requested vote from peers 4290f95cc39945379be798778eae2e18 (127.0.97.2:42859), 7fa1b50d6220465ab5e6409556e6ee7e (127.0.97.3:36767)
I20260620 23:36:25.159358 7774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc037ed734fd4843899753f7dbc8f863" candidate_uuid: "c6c5446de0824443bba78a92059589a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa1b50d6220465ab5e6409556e6ee7e"
I20260620 23:36:25.159430 7633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc037ed734fd4843899753f7dbc8f863" candidate_uuid: "c6c5446de0824443bba78a92059589a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4290f95cc39945379be798778eae2e18"
I20260620 23:36:25.159608 7774 raft_consensus.cc:3037] T bc037ed734fd4843899753f7dbc8f863 P 7fa1b50d6220465ab5e6409556e6ee7e [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:36:25.159699 7633 raft_consensus.cc:3037] T bc037ed734fd4843899753f7dbc8f863 P 4290f95cc39945379be798778eae2e18 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:36:25.162838 7774 raft_consensus.cc:2454] T bc037ed734fd4843899753f7dbc8f863 P 7fa1b50d6220465ab5e6409556e6ee7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6c5446de0824443bba78a92059589a9 in term 1.
I20260620 23:36:25.162838 7633 raft_consensus.cc:2454] T bc037ed734fd4843899753f7dbc8f863 P 4290f95cc39945379be798778eae2e18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6c5446de0824443bba78a92059589a9 in term 1.
I20260620 23:36:25.163319 7419 leader_election.cc:302] T bc037ed734fd4843899753f7dbc8f863 P c6c5446de0824443bba78a92059589a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4290f95cc39945379be798778eae2e18, 7fa1b50d6220465ab5e6409556e6ee7e, c6c5446de0824443bba78a92059589a9; no voters:
I20260620 23:36:25.163666 7863 raft_consensus.cc:2781] T bc037ed734fd4843899753f7dbc8f863 P c6c5446de0824443bba78a92059589a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:36:25.164171 7863 raft_consensus.cc:686] T bc037ed734fd4843899753f7dbc8f863 P c6c5446de0824443bba78a92059589a9 [term 1 LEADER]: Becoming Leader. State: Replica: c6c5446de0824443bba78a92059589a9, State: Running, Role: LEADER
I20260620 23:36:25.164672 7863 consensus_queue.cc:237] T bc037ed734fd4843899753f7dbc8f863 P c6c5446de0824443bba78a92059589a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6c5446de0824443bba78a92059589a9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45847 } } peers { permanent_uuid: "4290f95cc39945379be798778eae2e18" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42859 } } peers { permanent_uuid: "7fa1b50d6220465ab5e6409556e6ee7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36767 } }
I20260620 23:36:25.169484 6557 catalog_manager.cc:5495] T bc037ed734fd4843899753f7dbc8f863 P c6c5446de0824443bba78a92059589a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6c5446de0824443bba78a92059589a9 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "c6c5446de0824443bba78a92059589a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6c5446de0824443bba78a92059589a9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45847 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4290f95cc39945379be798778eae2e18" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fa1b50d6220465ab5e6409556e6ee7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36767 } health_report { overall_health: UNKNOWN } } }
I20260620 23:36:25.210246 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:25.204848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55326 (local address 127.0.97.1:45847)
0620 23:36:25.205044 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:25.205049 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:25.205067 (+ 18us) server_negotiation.cc:407] Connection header received
0620 23:36:25.205231 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:25.205235 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:25.205323 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:25.205453 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:36:25.205764 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:25.206455 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:25.207927 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:25.208349 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:25.208510 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:25.208987 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:36:25.209502 (+ 515us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:36:25.209698 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:25.209872 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 23:36:25.209960 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 23:36:25.219138 7774 raft_consensus.cc:1261] T bc037ed734fd4843899753f7dbc8f863 P 7fa1b50d6220465ab5e6409556e6ee7e [term 1 FOLLOWER]: Refusing update from remote peer c6c5446de0824443bba78a92059589a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:36:25.219159 7633 raft_consensus.cc:1261] T bc037ed734fd4843899753f7dbc8f863 P 4290f95cc39945379be798778eae2e18 [term 1 FOLLOWER]: Refusing update from remote peer c6c5446de0824443bba78a92059589a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:36:25.220319 7867 consensus_queue.cc:1035] T bc037ed734fd4843899753f7dbc8f863 P c6c5446de0824443bba78a92059589a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa1b50d6220465ab5e6409556e6ee7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:36:25.220729 7863 consensus_queue.cc:1035] T bc037ed734fd4843899753f7dbc8f863 P c6c5446de0824443bba78a92059589a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4290f95cc39945379be798778eae2e18" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:36:25.233317 7871 mvcc.cc:205] Tried to move back new op lower bound from 7299066205046280192 to 7299066204838072320. Current Snapshot: MvccSnapshot[applied={T|T < 7299066205046280192}]
I20260620 23:36:25.242312 7873 mvcc.cc:205] Tried to move back new op lower bound from 7299066205046280192 to 7299066204838072320. Current Snapshot: MvccSnapshot[applied={T|T < 7299066205046280192}]
I20260620 23:36:25.256299 6557 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:36:25.256593 6557 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:36:25.262825 6557 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9d4ee78c582c4c9c9ae36386fc5afcaf: Sending DeleteTablet for 3 replicas of tablet bc037ed734fd4843899753f7dbc8f863
I20260620 23:36:25.264391 7465 tablet_service.cc:1514] Processing DeleteTablet for tablet bc037ed734fd4843899753f7dbc8f863 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:36:25 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55320
I20260620 23:36:25.265231 7754 tablet_service.cc:1514] Processing DeleteTablet for tablet bc037ed734fd4843899753f7dbc8f863 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:36:25 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58374
I20260620 23:36:25.266670 7613 tablet_service.cc:1514] Processing DeleteTablet for tablet bc037ed734fd4843899753f7dbc8f863 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:36:25 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55884
I20260620 23:36:25.269411 7882 tablet_replica.cc:330] T bc037ed734fd4843899753f7dbc8f863 P 7fa1b50d6220465ab5e6409556e6ee7e: stopping tablet replica
I20260620 23:36:25.270022 7882 raft_consensus.cc:2229] T bc037ed734fd4843899753f7dbc8f863 P 7fa1b50d6220465ab5e6409556e6ee7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:36:25.270687 7882 raft_consensus.cc:2258] T bc037ed734fd4843899753f7dbc8f863 P 7fa1b50d6220465ab5e6409556e6ee7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:36:25.271623 7883 tablet_replica.cc:330] T bc037ed734fd4843899753f7dbc8f863 P 4290f95cc39945379be798778eae2e18: stopping tablet replica
I20260620 23:36:25.272202 7883 raft_consensus.cc:2229] T bc037ed734fd4843899753f7dbc8f863 P 4290f95cc39945379be798778eae2e18 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:36:25.272847 7881 tablet_replica.cc:330] T bc037ed734fd4843899753f7dbc8f863 P c6c5446de0824443bba78a92059589a9: stopping tablet replica
I20260620 23:36:25.273352 7883 raft_consensus.cc:2258] T bc037ed734fd4843899753f7dbc8f863 P 4290f95cc39945379be798778eae2e18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:36:25.273828 7881 raft_consensus.cc:2229] T bc037ed734fd4843899753f7dbc8f863 P c6c5446de0824443bba78a92059589a9 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:36:25.274580 7882 ts_tablet_manager.cc:1905] T bc037ed734fd4843899753f7dbc8f863 P 7fa1b50d6220465ab5e6409556e6ee7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:36:25.276607 7883 ts_tablet_manager.cc:1905] T bc037ed734fd4843899753f7dbc8f863 P 4290f95cc39945379be798778eae2e18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:36:25.276829 7881 raft_consensus.cc:2258] T bc037ed734fd4843899753f7dbc8f863 P c6c5446de0824443bba78a92059589a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:36:25.278821 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 6527
I20260620 23:36:25.280071 6589 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:36:25.283754 7881 ts_tablet_manager.cc:1905] T bc037ed734fd4843899753f7dbc8f863 P c6c5446de0824443bba78a92059589a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:36:25.287331 7883 ts_tablet_manager.cc:1918] T bc037ed734fd4843899753f7dbc8f863 P 4290f95cc39945379be798778eae2e18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:36:25.287523 7883 log.cc:1196] T bc037ed734fd4843899753f7dbc8f863 P 4290f95cc39945379be798778eae2e18: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-1/wal/wals/bc037ed734fd4843899753f7dbc8f863
I20260620 23:36:25.288111 7883 ts_tablet_manager.cc:1939] T bc037ed734fd4843899753f7dbc8f863 P 4290f95cc39945379be798778eae2e18: Deleting consensus metadata
I20260620 23:36:25.295948 7881 ts_tablet_manager.cc:1918] T bc037ed734fd4843899753f7dbc8f863 P c6c5446de0824443bba78a92059589a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:36:25.296178 7881 log.cc:1196] T bc037ed734fd4843899753f7dbc8f863 P c6c5446de0824443bba78a92059589a9: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-0/wal/wals/bc037ed734fd4843899753f7dbc8f863
I20260620 23:36:25.296139 7882 ts_tablet_manager.cc:1918] T bc037ed734fd4843899753f7dbc8f863 P 7fa1b50d6220465ab5e6409556e6ee7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:36:25.296324 7882 log.cc:1196] T bc037ed734fd4843899753f7dbc8f863 P 7fa1b50d6220465ab5e6409556e6ee7e: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-2/wal/wals/bc037ed734fd4843899753f7dbc8f863
I20260620 23:36:25.296836 7881 ts_tablet_manager.cc:1939] T bc037ed734fd4843899753f7dbc8f863 P c6c5446de0824443bba78a92059589a9: Deleting consensus metadata
I20260620 23:36:25.296850 7882 ts_tablet_manager.cc:1939] T bc037ed734fd4843899753f7dbc8f863 P 7fa1b50d6220465ab5e6409556e6ee7e: Deleting consensus metadata
I20260620 23:36:25.308040 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:25.267616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56486 (local address 127.0.97.61:39051)
0620 23:36:25.267976 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:25.267983 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:25.277829 (+ 9846us) server_negotiation.cc:407] Connection header received
0620 23:36:25.278323 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:25.278329 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:25.278420 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:25.278526 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:25.291764 (+ 13238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:25.292896 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:25.297667 (+ 4771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:25.298299 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:25.298415 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:25.301694 (+ 3279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:25.301699 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:25.301702 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:25.301738 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:25.304570 (+ 2832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:25.305563 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:25.305569 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:25.305572 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:25.305718 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:25.306177 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:25.306182 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:25.306185 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:25.306533 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:25.306770 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:25.307445 (+ 675us) server_negotiation.cc:299] Negotiation successful
0620 23:36:25.307693 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221}
I20260620 23:36:25.328224 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:25.277997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34176 (local address 127.0.97.60:33735)
0620 23:36:25.278212 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:25.278219 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:25.310837 (+ 32618us) server_negotiation.cc:407] Connection header received
0620 23:36:25.311163 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:25.311170 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:25.311268 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:25.311334 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:25.313196 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:25.314336 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:25.316480 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:25.317143 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:25.317260 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:25.320993 (+ 3733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:25.321000 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:25.321005 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:25.321056 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:25.324670 (+ 3614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:25.325617 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:25.325624 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:25.325628 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:25.325783 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:25.326253 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:25.326261 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:25.326266 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:25.326583 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:25.326753 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:25.327636 (+ 883us) server_negotiation.cc:299] Negotiation successful
0620 23:36:25.327876 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 23:36:25.441169 6544 catalog_manager.cc:4841] TS 7fa1b50d6220465ab5e6409556e6ee7e (127.0.97.3:36767): tablet bc037ed734fd4843899753f7dbc8f863 (table test-table [id=9c24694f672546c29a12f2ece3b0fffb]) successfully deleted
I20260620 23:36:25.442019 6546 catalog_manager.cc:4841] TS 4290f95cc39945379be798778eae2e18 (127.0.97.2:42859): tablet bc037ed734fd4843899753f7dbc8f863 (table test-table [id=9c24694f672546c29a12f2ece3b0fffb]) successfully deleted
I20260620 23:36:25.442296 6544 catalog_manager.cc:4841] TS c6c5446de0824443bba78a92059589a9 (127.0.97.1:45847): tablet bc037ed734fd4843899753f7dbc8f863 (table test-table [id=9c24694f672546c29a12f2ece3b0fffb]) successfully deleted
I20260620 23:36:25.442730 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 6527
I20260620 23:36:25.505564 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:25.483869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33722 (local address 127.0.97.60:33735)
0620 23:36:25.484074 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:25.484080 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:25.488142 (+ 4062us) server_negotiation.cc:407] Connection header received
0620 23:36:25.488545 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:25.488550 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:25.488629 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:25.488696 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:25.491153 (+ 2457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:25.492311 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:25.494471 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:25.495086 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:25.495238 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:25.499193 (+ 3955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:25.499202 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:25.499208 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:25.499261 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:25.502502 (+ 3241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:25.503339 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:25.503344 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:25.503346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:25.503467 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:25.503955 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:25.503960 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:25.503964 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:25.504336 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:25.504592 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:25.505026 (+ 434us) server_negotiation.cc:299] Negotiation successful
0620 23:36:25.505243 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 23:36:25.507947 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:25.484065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56448 (local address 127.0.97.61:39051)
0620 23:36:25.484238 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:25.484244 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:25.488179 (+ 3935us) server_negotiation.cc:407] Connection header received
0620 23:36:25.488489 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:25.488498 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:25.488599 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:25.488678 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:25.491423 (+ 2745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:25.492153 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:25.494405 (+ 2252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:25.495255 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:25.495412 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:25.500051 (+ 4639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:25.500057 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:25.500061 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:25.500102 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:25.503407 (+ 3305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:25.504196 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:25.504201 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:25.504205 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:25.504340 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:25.504784 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:25.504790 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:25.504794 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:25.505159 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:25.505552 (+ 393us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:25.507110 (+ 1558us) server_negotiation.cc:299] Negotiation successful
0620 23:36:25.507440 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 23:36:26.242914 7893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:26.242050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33681 (local address 127.0.97.3:58701)
0620 23:36:26.242527 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:26.242650 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":260,"threads_started":1}
W20260620 23:36:26.243331 7820 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:33681 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:33681: connect: Connection refused (error 111)
I20260620 23:36:26.244453 7894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:26.243744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33681 (local address 127.0.97.1:44463)
0620 23:36:26.244206 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:26.244304 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":252,"threads_started":1}
W20260620 23:36:26.244894 7531 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:33681 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:33681: connect: Connection refused (error 111)
I20260620 23:36:26.248870 7895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:26.248186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33681 (local address 127.0.97.2:34887)
0620 23:36:26.248603 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:26.248699 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":207,"threads_started":1}
W20260620 23:36:26.249310 7679 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:33681 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:33681: connect: Connection refused (error 111)
I20260620 23:36:26.763473 7896 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b672bfe633a443591fc2dd3ca7277b1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9d4ee78c582c4c9c9ae36386fc5afcaf)
I20260620 23:36:26.763790 7896 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b672bfe633a443591fc2dd3ca7277b1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33681 } } peers { permanent_uuid: "84fa2fff77be49b098f7d05dd4ec1a4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39051 } attrs { promote: false } } peers { permanent_uuid: "4b672bfe633a443591fc2dd3ca7277b1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33735 } attrs { promote: false } }
I20260620 23:36:26.764602 7897 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84fa2fff77be49b098f7d05dd4ec1a4a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9d4ee78c582c4c9c9ae36386fc5afcaf)
I20260620 23:36:26.764890 7897 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84fa2fff77be49b098f7d05dd4ec1a4a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33681 } } peers { permanent_uuid: "84fa2fff77be49b098f7d05dd4ec1a4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39051 } attrs { promote: false } } peers { permanent_uuid: "4b672bfe633a443591fc2dd3ca7277b1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33735 } attrs { promote: false } }
I20260620 23:36:26.765511 7896 leader_election.cc:288] T 00000000000000000000000000000000 P 4b672bfe633a443591fc2dd3ca7277b1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9d4ee78c582c4c9c9ae36386fc5afcaf (127.0.97.62:33681), 84fa2fff77be49b098f7d05dd4ec1a4a (127.0.97.61:39051)
I20260620 23:36:26.772681 7897 leader_election.cc:288] T 00000000000000000000000000000000 P 84fa2fff77be49b098f7d05dd4ec1a4a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9d4ee78c582c4c9c9ae36386fc5afcaf (127.0.97.62:33681), 4b672bfe633a443591fc2dd3ca7277b1 (127.0.97.60:33735)
I20260620 23:36:26.775398 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:26.765614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33681 (local address 127.0.0.1:50312)
0620 23:36:26.774767 (+ 9153us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:26.775072 (+ 305us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":8928,"threads_started":1}
W20260620 23:36:26.776513 7335 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33681: connect: Connection refused (error 111)
I20260620 23:36:26.778209 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:26.766625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33681 (local address 127.0.0.1:50322)
0620 23:36:26.777636 (+ 11011us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:26.777870 (+ 234us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":10839,"thread_start_us":10638,"threads_started":1}
W20260620 23:36:26.782101 7144 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33681: connect: Connection refused (error 111)
I20260620 23:36:26.785583 7899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:26.784958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33681 (local address 127.0.0.1:50324)
0620 23:36:26.785158 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:26.785340 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 23:36:26.785827 7335 leader_election.cc:334] T 00000000000000000000000000000000 P 4b672bfe633a443591fc2dd3ca7277b1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9d4ee78c582c4c9c9ae36386fc5afcaf (127.0.97.62:33681): Network error: Client connection negotiation failed: client connection to 127.0.97.62:33681: connect: Connection refused (error 111)
I20260620 23:36:26.787360 7900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:26.765906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56452 (local address 127.0.97.61:39051)
0620 23:36:26.777966 (+ 12060us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:26.777975 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:26.778004 (+ 29us) server_negotiation.cc:407] Connection header received
0620 23:36:26.778106 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:26.778115 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:26.778228 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:26.778374 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:36:26.779352 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:26.781030 (+ 1678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:26.784194 (+ 3164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:26.785803 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:26.785939 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:26.786047 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:26.786603 (+ 556us) server_negotiation.cc:299] Negotiation successful
0620 23:36:26.786748 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11904,"thread_start_us":12578,"threads_started":1}
I20260620 23:36:26.787487 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:26.765767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39051 (local address 127.0.0.1:56452)
0620 23:36:26.775656 (+ 9889us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:26.775682 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:26.775891 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:26.778627 (+ 2736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:26.778636 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:26.778687 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:36:26.779106 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:26.779125 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:26.781296 (+ 2171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:26.781307 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:26.783923 (+ 2616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:26.783947 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:26.786213 (+ 2266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:26.786219 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:26.786421 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:26.786461 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 23:36:26.786995 (+ 534us) client_negotiation.cc:240] Negotiation successful
0620 23:36:26.787165 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9771,"thread_start_us":15842,"threads_started":1}
I20260620 23:36:26.788106 7903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:26.786636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33681 (local address 127.0.0.1:50334)
0620 23:36:26.787718 (+ 1082us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:26.787857 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":969}
W20260620 23:36:26.788266 7144 leader_election.cc:334] T 00000000000000000000000000000000 P 84fa2fff77be49b098f7d05dd4ec1a4a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9d4ee78c582c4c9c9ae36386fc5afcaf (127.0.97.62:33681): Network error: Client connection negotiation failed: client connection to 127.0.97.62:33681: connect: Connection refused (error 111)
I20260620 23:36:26.788414 7170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4b672bfe633a443591fc2dd3ca7277b1" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "84fa2fff77be49b098f7d05dd4ec1a4a" is_pre_election: true
I20260620 23:36:26.788789 7170 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 84fa2fff77be49b098f7d05dd4ec1a4a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b672bfe633a443591fc2dd3ca7277b1 in term 1.
I20260620 23:36:26.789331 7338 leader_election.cc:302] T 00000000000000000000000000000000 P 4b672bfe633a443591fc2dd3ca7277b1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4b672bfe633a443591fc2dd3ca7277b1, 84fa2fff77be49b098f7d05dd4ec1a4a; no voters: 9d4ee78c582c4c9c9ae36386fc5afcaf
I20260620 23:36:26.789706 7896 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b672bfe633a443591fc2dd3ca7277b1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 23:36:26.789826 7896 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b672bfe633a443591fc2dd3ca7277b1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9d4ee78c582c4c9c9ae36386fc5afcaf)
I20260620 23:36:26.789911 7896 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b672bfe633a443591fc2dd3ca7277b1 [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:36:26.792814 7896 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b672bfe633a443591fc2dd3ca7277b1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33681 } } peers { permanent_uuid: "84fa2fff77be49b098f7d05dd4ec1a4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39051 } attrs { promote: false } } peers { permanent_uuid: "4b672bfe633a443591fc2dd3ca7277b1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33735 } attrs { promote: false } }
I20260620 23:36:26.793092 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:26.779374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33735 (local address 127.0.0.1:33738)
0620 23:36:26.786350 (+ 6976us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:26.786370 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:26.786538 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:26.787104 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:26.787108 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:26.787134 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:36:26.787429 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:26.787440 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:26.788822 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:26.788830 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:26.790627 (+ 1797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:26.790639 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:26.792343 (+ 1704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:26.792351 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:26.792487 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:26.792511 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 23:36:26.792621 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 23:36:26.792766 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6870,"thread_start_us":8348,"threads_started":1}
I20260620 23:36:26.793196 7904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:26.779397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33738 (local address 127.0.97.60:33735)
0620 23:36:26.781629 (+ 2232us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:26.781637 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:26.786441 (+ 4804us) server_negotiation.cc:407] Connection header received
0620 23:36:26.786693 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:26.786700 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:26.786768 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:26.786835 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:36:26.787748 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:26.788702 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:26.790779 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:26.791873 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:26.792209 (+ 336us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:26.792338 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:26.792645 (+ 307us) server_negotiation.cc:299] Negotiation successful
0620 23:36:26.792776 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2030,"thread_start_us":1817,"threads_started":1}
I20260620 23:36:26.793648 7896 leader_election.cc:288] T 00000000000000000000000000000000 P 4b672bfe633a443591fc2dd3ca7277b1 [CANDIDATE]: Term 2 election: Requested vote from peers 9d4ee78c582c4c9c9ae36386fc5afcaf (127.0.97.62:33681), 84fa2fff77be49b098f7d05dd4ec1a4a (127.0.97.61:39051)
I20260620 23:36:26.793887 7361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "84fa2fff77be49b098f7d05dd4ec1a4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4b672bfe633a443591fc2dd3ca7277b1" is_pre_election: true
I20260620 23:36:26.794678 7361 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 4b672bfe633a443591fc2dd3ca7277b1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 84fa2fff77be49b098f7d05dd4ec1a4a in current term 2: Already voted for candidate 4b672bfe633a443591fc2dd3ca7277b1 in this term.
I20260620 23:36:26.794725 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:26.793748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33681 (local address 127.0.0.1:50354)
0620 23:36:26.793959 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:26.794043 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260620 23:36:26.795174 7170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4b672bfe633a443591fc2dd3ca7277b1" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "84fa2fff77be49b098f7d05dd4ec1a4a"
I20260620 23:36:26.795457 7170 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84fa2fff77be49b098f7d05dd4ec1a4a [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:36:26.795796 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:26.795495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33681 (local address 127.0.0.1:50358)
0620 23:36:26.795629 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:26.795712 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 23:36:26.795950 7335 leader_election.cc:334] T 00000000000000000000000000000000 P 4b672bfe633a443591fc2dd3ca7277b1 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9d4ee78c582c4c9c9ae36386fc5afcaf (127.0.97.62:33681): Network error: Client connection negotiation failed: client connection to 127.0.97.62:33681: connect: Connection refused (error 111)
I20260620 23:36:26.797607 7170 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 84fa2fff77be49b098f7d05dd4ec1a4a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b672bfe633a443591fc2dd3ca7277b1 in term 2.
I20260620 23:36:26.797966 7338 leader_election.cc:302] T 00000000000000000000000000000000 P 4b672bfe633a443591fc2dd3ca7277b1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4b672bfe633a443591fc2dd3ca7277b1, 84fa2fff77be49b098f7d05dd4ec1a4a; no voters: 9d4ee78c582c4c9c9ae36386fc5afcaf
I20260620 23:36:26.798185 7145 leader_election.cc:302] T 00000000000000000000000000000000 P 84fa2fff77be49b098f7d05dd4ec1a4a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84fa2fff77be49b098f7d05dd4ec1a4a; no voters: 4b672bfe633a443591fc2dd3ca7277b1, 9d4ee78c582c4c9c9ae36386fc5afcaf
I20260620 23:36:26.798276 7896 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b672bfe633a443591fc2dd3ca7277b1 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 23:36:26.798566 7897 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 84fa2fff77be49b098f7d05dd4ec1a4a [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:36:26.798810 7896 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b672bfe633a443591fc2dd3ca7277b1 [term 2 LEADER]: Becoming Leader. State: Replica: 4b672bfe633a443591fc2dd3ca7277b1, State: Running, Role: LEADER
I20260620 23:36:26.799221 7896 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b672bfe633a443591fc2dd3ca7277b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33681 } } peers { permanent_uuid: "84fa2fff77be49b098f7d05dd4ec1a4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39051 } attrs { promote: false } } peers { permanent_uuid: "4b672bfe633a443591fc2dd3ca7277b1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33735 } attrs { promote: false } }
I20260620 23:36:26.800978 7906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b672bfe633a443591fc2dd3ca7277b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b672bfe633a443591fc2dd3ca7277b1. Latest consensus state: current_term: 2 leader_uuid: "4b672bfe633a443591fc2dd3ca7277b1" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33681 } } peers { permanent_uuid: "84fa2fff77be49b098f7d05dd4ec1a4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39051 } attrs { promote: false } } peers { permanent_uuid: "4b672bfe633a443591fc2dd3ca7277b1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33735 } attrs { promote: false } } }
I20260620 23:36:26.801393 7906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b672bfe633a443591fc2dd3ca7277b1 [sys.catalog]: This master's current role is: LEADER
I20260620 23:36:26.802042 7908 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:36:26.805589 7908 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:36:26.806413 7908 catalog_manager.cc:1259] Loaded cluster ID: 7fb67d48eb2a4aebad7304834a960bf3
I20260620 23:36:26.806486 7908 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:36:26.807746 7908 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:36:26.808540 7908 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4b672bfe633a443591fc2dd3ca7277b1: Loaded TSK: 0
I20260620 23:36:26.809423 7908 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:36:26.855223 7351 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33722:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:36:26.862702 7351 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:36:26.867595 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:26.867068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33681 (local address 127.0.0.1:50370)
0620 23:36:26.867243 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:26.867373 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 23:36:26.867646 7170 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 84fa2fff77be49b098f7d05dd4ec1a4a [term 2 FOLLOWER]: Refusing update from remote peer 4b672bfe633a443591fc2dd3ca7277b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260620 23:36:26.868428 7906 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4b672bfe633a443591fc2dd3ca7277b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84fa2fff77be49b098f7d05dd4ec1a4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39051 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260620 23:36:26.868855 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:26.868462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33681 (local address 127.0.0.1:50384)
0620 23:36:26.868620 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:26.868705 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260620 23:36:26.869088 7335 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4b672bfe633a443591fc2dd3ca7277b1 -> Peer 9d4ee78c582c4c9c9ae36386fc5afcaf (127.0.97.62:33681): Couldn't send request to peer 9d4ee78c582c4c9c9ae36386fc5afcaf. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33681: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 23:36:26.871439 7897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84fa2fff77be49b098f7d05dd4ec1a4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b672bfe633a443591fc2dd3ca7277b1. Latest consensus state: current_term: 2 leader_uuid: "4b672bfe633a443591fc2dd3ca7277b1" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33681 } } peers { permanent_uuid: "84fa2fff77be49b098f7d05dd4ec1a4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39051 } attrs { promote: false } } peers { permanent_uuid: "4b672bfe633a443591fc2dd3ca7277b1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33735 } attrs { promote: false } } }
I20260620 23:36:26.871768 7897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84fa2fff77be49b098f7d05dd4ec1a4a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:36:26.872501 7909 mvcc.cc:205] Tried to move back new op lower bound from 7299066211802144768 to 7299066211532902400. Current Snapshot: MvccSnapshot[applied={T|T < 7299066211802144768}]
I20260620 23:36:26.879546 7897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84fa2fff77be49b098f7d05dd4ec1a4a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "4b672bfe633a443591fc2dd3ca7277b1" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33681 } } peers { permanent_uuid: "84fa2fff77be49b098f7d05dd4ec1a4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39051 } attrs { promote: false } } peers { permanent_uuid: "4b672bfe633a443591fc2dd3ca7277b1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33735 } attrs { promote: false } } }
I20260620 23:36:26.880172 7897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84fa2fff77be49b098f7d05dd4ec1a4a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:36:26.881625 7896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b672bfe633a443591fc2dd3ca7277b1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "4b672bfe633a443591fc2dd3ca7277b1" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9d4ee78c582c4c9c9ae36386fc5afcaf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33681 } } peers { permanent_uuid: "84fa2fff77be49b098f7d05dd4ec1a4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39051 } attrs { promote: false } } peers { permanent_uuid: "4b672bfe633a443591fc2dd3ca7277b1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33735 } attrs { promote: false } } }
I20260620 23:36:26.881974 7896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b672bfe633a443591fc2dd3ca7277b1 [sys.catalog]: This master's current role is: LEADER
I20260620 23:36:26.909541 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:26.890368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42859 (local address 127.0.0.1:47012)
0620 23:36:26.890522 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:26.890555 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:26.890766 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:26.904288 (+ 13522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:26.904293 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:26.904323 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:36:26.904531 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:26.904542 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:26.905835 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:26.905840 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:26.907617 (+ 1777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:26.907632 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:26.908674 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:26.908680 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:26.908883 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:26.908920 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 23:36:26.909055 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 23:36:26.909203 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":27}
I20260620 23:36:26.909508 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:26.890492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47012 (local address 127.0.97.2:42859)
0620 23:36:26.903977 (+ 13485us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:26.903982 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:26.903999 (+ 17us) server_negotiation.cc:407] Connection header received
0620 23:36:26.904058 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:26.904062 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:26.904120 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:26.904255 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:36:26.904717 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:26.905684 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:26.907775 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:26.908518 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:26.908682 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:26.908865 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:26.909076 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 23:36:26.909212 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13368,"thread_start_us":13278,"threads_started":1}
I20260620 23:36:26.910655 7613 tablet_service.cc:1467] Processing CreateTablet for tablet 006d7fd72c7d44e7a9459887c0cf3771 (DEFAULT_TABLE table=test-table [id=ed5035d48f8041a196c650ae242b32aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:36:26.911145 7613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 006d7fd72c7d44e7a9459887c0cf3771. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:36:26.912279 7899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:26.890820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36767 (local address 127.0.0.1:48502)
0620 23:36:26.891139 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:26.891159 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:26.891273 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:26.904753 (+ 13480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:26.904759 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:26.904793 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:36:26.905106 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:26.905120 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:26.906846 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:26.906853 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:26.909447 (+ 2594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:26.909461 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:26.911279 (+ 1818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:26.911286 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:26.911517 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:26.911563 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 23:36:26.911630 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 23:36:26.911900 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":92}
I20260620 23:36:26.912660 7918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:26.891653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48502 (local address 127.0.97.3:36767)
0620 23:36:26.903800 (+ 12147us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:26.903811 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:26.903843 (+ 32us) server_negotiation.cc:407] Connection header received
0620 23:36:26.903941 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:26.903948 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:26.904071 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:26.904275 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:36:26.905688 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:26.906705 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:26.909674 (+ 2969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:26.910989 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:26.911841 (+ 852us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:26.911962 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:26.912020 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 23:36:26.912174 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11963,"thread_start_us":11809,"threads_started":1}
I20260620 23:36:26.913331 7754 tablet_service.cc:1467] Processing CreateTablet for tablet 006d7fd72c7d44e7a9459887c0cf3771 (DEFAULT_TABLE table=test-table [id=ed5035d48f8041a196c650ae242b32aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:36:26.913792 7754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 006d7fd72c7d44e7a9459887c0cf3771. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:36:26.915841 7917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:26.891398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37278 (local address 127.0.97.1:45847)
0620 23:36:26.909024 (+ 17626us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:26.909032 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:26.909053 (+ 21us) server_negotiation.cc:407] Connection header received
0620 23:36:26.909117 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:26.909122 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:26.909190 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:26.909287 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:36:26.910106 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:26.911177 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:26.913571 (+ 2394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:26.914849 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:26.915053 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:26.915168 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:26.915391 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 23:36:26.915549 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17492,"thread_start_us":17956,"threads_started":1}
I20260620 23:36:26.915848 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:26.891183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45847 (local address 127.0.0.1:37278)
0620 23:36:26.906167 (+ 14984us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:26.906188 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:26.906302 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:26.909540 (+ 3238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:26.909545 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:26.909586 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:36:26.909933 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:26.909943 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:26.911316 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:26.911321 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:26.913328 (+ 2007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:26.913342 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:26.915047 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:26.915053 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:26.915224 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:26.915257 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 23:36:26.915352 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 23:36:26.915490 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14831,"thread_start_us":15158,"threads_started":1}
I20260620 23:36:26.916859 7465 tablet_service.cc:1467] Processing CreateTablet for tablet 006d7fd72c7d44e7a9459887c0cf3771 (DEFAULT_TABLE table=test-table [id=ed5035d48f8041a196c650ae242b32aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:36:26.917317 7465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 006d7fd72c7d44e7a9459887c0cf3771. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:36:26.918237 7919 tablet_bootstrap.cc:492] T 006d7fd72c7d44e7a9459887c0cf3771 P 4290f95cc39945379be798778eae2e18: Bootstrap starting.
I20260620 23:36:26.920210 7919 tablet_bootstrap.cc:654] T 006d7fd72c7d44e7a9459887c0cf3771 P 4290f95cc39945379be798778eae2e18: Neither blocks nor log segments found. Creating new log.
I20260620 23:36:26.921303 7920 tablet_bootstrap.cc:492] T 006d7fd72c7d44e7a9459887c0cf3771 P 7fa1b50d6220465ab5e6409556e6ee7e: Bootstrap starting.
I20260620 23:36:26.923040 7920 tablet_bootstrap.cc:654] T 006d7fd72c7d44e7a9459887c0cf3771 P 7fa1b50d6220465ab5e6409556e6ee7e: Neither blocks nor log segments found. Creating new log.
I20260620 23:36:26.923614 7919 tablet_bootstrap.cc:492] T 006d7fd72c7d44e7a9459887c0cf3771 P 4290f95cc39945379be798778eae2e18: No bootstrap required, opened a new log
I20260620 23:36:26.923781 7919 ts_tablet_manager.cc:1397] T 006d7fd72c7d44e7a9459887c0cf3771 P 4290f95cc39945379be798778eae2e18: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260620 23:36:26.924609 7919 raft_consensus.cc:348] T 006d7fd72c7d44e7a9459887c0cf3771 P 4290f95cc39945379be798778eae2e18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4290f95cc39945379be798778eae2e18" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42859 } } peers { permanent_uuid: "7fa1b50d6220465ab5e6409556e6ee7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36767 } } peers { permanent_uuid: "c6c5446de0824443bba78a92059589a9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45847 } }
I20260620 23:36:26.924877 7919 raft_consensus.cc:374] T 006d7fd72c7d44e7a9459887c0cf3771 P 4290f95cc39945379be798778eae2e18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:36:26.924965 7919 raft_consensus.cc:729] T 006d7fd72c7d44e7a9459887c0cf3771 P 4290f95cc39945379be798778eae2e18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4290f95cc39945379be798778eae2e18, State: Initialized, Role: FOLLOWER
I20260620 23:36:26.925208 7919 consensus_queue.cc:260] T 006d7fd72c7d44e7a9459887c0cf3771 P 4290f95cc39945379be798778eae2e18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4290f95cc39945379be798778eae2e18" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42859 } } peers { permanent_uuid: "7fa1b50d6220465ab5e6409556e6ee7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36767 } } peers { permanent_uuid: "c6c5446de0824443bba78a92059589a9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45847 } }
I20260620 23:36:26.925555 7920 tablet_bootstrap.cc:492] T 006d7fd72c7d44e7a9459887c0cf3771 P 7fa1b50d6220465ab5e6409556e6ee7e: No bootstrap required, opened a new log
I20260620 23:36:26.925746 7920 ts_tablet_manager.cc:1397] T 006d7fd72c7d44e7a9459887c0cf3771 P 7fa1b50d6220465ab5e6409556e6ee7e: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260620 23:36:26.926052 7919 ts_tablet_manager.cc:1428] T 006d7fd72c7d44e7a9459887c0cf3771 P 4290f95cc39945379be798778eae2e18: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 23:36:26.929013 7922 tablet_bootstrap.cc:492] T 006d7fd72c7d44e7a9459887c0cf3771 P c6c5446de0824443bba78a92059589a9: Bootstrap starting.
I20260620 23:36:26.929967 7678 heartbeater.cc:503] Master 127.0.97.60:33735 was elected leader, sending a full tablet report...
I20260620 23:36:26.930823 7922 tablet_bootstrap.cc:654] T 006d7fd72c7d44e7a9459887c0cf3771 P c6c5446de0824443bba78a92059589a9: Neither blocks nor log segments found. Creating new log.
I20260620 23:36:26.933060 7920 raft_consensus.cc:348] T 006d7fd72c7d44e7a9459887c0cf3771 P 7fa1b50d6220465ab5e6409556e6ee7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4290f95cc39945379be798778eae2e18" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42859 } } peers { permanent_uuid: "7fa1b50d6220465ab5e6409556e6ee7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36767 } } peers { permanent_uuid: "c6c5446de0824443bba78a92059589a9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45847 } }
I20260620 23:36:26.933314 7920 raft_consensus.cc:374] T 006d7fd72c7d44e7a9459887c0cf3771 P 7fa1b50d6220465ab5e6409556e6ee7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:36:26.933411 7920 raft_consensus.cc:729] T 006d7fd72c7d44e7a9459887c0cf3771 P 7fa1b50d6220465ab5e6409556e6ee7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa1b50d6220465ab5e6409556e6ee7e, State: Initialized, Role: FOLLOWER
I20260620 23:36:26.933655 7922 tablet_bootstrap.cc:492] T 006d7fd72c7d44e7a9459887c0cf3771 P c6c5446de0824443bba78a92059589a9: No bootstrap required, opened a new log
I20260620 23:36:26.933795 7922 ts_tablet_manager.cc:1397] T 006d7fd72c7d44e7a9459887c0cf3771 P c6c5446de0824443bba78a92059589a9: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 23:36:26.933712 7920 consensus_queue.cc:260] T 006d7fd72c7d44e7a9459887c0cf3771 P 7fa1b50d6220465ab5e6409556e6ee7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4290f95cc39945379be798778eae2e18" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42859 } } peers { permanent_uuid: "7fa1b50d6220465ab5e6409556e6ee7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36767 } } peers { permanent_uuid: "c6c5446de0824443bba78a92059589a9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45847 } }
I20260620 23:36:26.934537 7920 ts_tablet_manager.cc:1428] T 006d7fd72c7d44e7a9459887c0cf3771 P 7fa1b50d6220465ab5e6409556e6ee7e: Time spent starting tablet: real 0.009s user 0.000s sys 0.003s
I20260620 23:36:26.934626 7922 raft_consensus.cc:348] T 006d7fd72c7d44e7a9459887c0cf3771 P c6c5446de0824443bba78a92059589a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4290f95cc39945379be798778eae2e18" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42859 } } peers { permanent_uuid: "7fa1b50d6220465ab5e6409556e6ee7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36767 } } peers { permanent_uuid: "c6c5446de0824443bba78a92059589a9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45847 } }
I20260620 23:36:26.934844 7922 raft_consensus.cc:374] T 006d7fd72c7d44e7a9459887c0cf3771 P c6c5446de0824443bba78a92059589a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:36:26.934962 7922 raft_consensus.cc:729] T 006d7fd72c7d44e7a9459887c0cf3771 P c6c5446de0824443bba78a92059589a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6c5446de0824443bba78a92059589a9, State: Initialized, Role: FOLLOWER
I20260620 23:36:26.935273 7922 consensus_queue.cc:260] T 006d7fd72c7d44e7a9459887c0cf3771 P c6c5446de0824443bba78a92059589a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4290f95cc39945379be798778eae2e18" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42859 } } peers { permanent_uuid: "7fa1b50d6220465ab5e6409556e6ee7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36767 } } peers { permanent_uuid: "c6c5446de0824443bba78a92059589a9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45847 } }
I20260620 23:36:26.937584 7819 heartbeater.cc:503] Master 127.0.97.60:33735 was elected leader, sending a full tablet report...
I20260620 23:36:26.941700 7928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:26.935191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33681 (local address 127.0.97.3:55341)
0620 23:36:26.941289 (+ 6098us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:26.941425 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5953,"threads_started":1}
I20260620 23:36:26.942437 7922 ts_tablet_manager.cc:1428] T 006d7fd72c7d44e7a9459887c0cf3771 P c6c5446de0824443bba78a92059589a9: Time spent starting tablet: real 0.009s user 0.001s sys 0.008s
I20260620 23:36:26.943962 7925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:26.926601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33681 (local address 127.0.97.2:57749)
0620 23:36:26.943486 (+ 16885us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:26.943634 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16749,"threads_started":1}
I20260620 23:36:26.944015 7930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:26.943147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33681 (local address 127.0.97.1:38115)
0620 23:36:26.943650 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:26.943777 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"threads_started":1}
I20260620 23:36:26.945433 7530 heartbeater.cc:503] Master 127.0.97.60:33735 was elected leader, sending a full tablet report...
I20260620 23:36:26.945886 7925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:26.945455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33681 (local address 127.0.97.2:38705)
0620 23:36:26.945662 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:26.945746 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260620 23:36:26.946086 7930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:26.945696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33681 (local address 127.0.97.1:35357)
0620 23:36:26.945861 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:26.945948 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 23:36:26.946465 7928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:26.946096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33681 (local address 127.0.97.3:41351)
0620 23:36:26.946244 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:26.946313 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 23:36:27.006863 7681 tablet.cc:2369] T 006d7fd72c7d44e7a9459887c0cf3771 P 4290f95cc39945379be798778eae2e18: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:36:27.066632 7929 raft_consensus.cc:482] T 006d7fd72c7d44e7a9459887c0cf3771 P c6c5446de0824443bba78a92059589a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:36:27.066958 7929 raft_consensus.cc:504] T 006d7fd72c7d44e7a9459887c0cf3771 P c6c5446de0824443bba78a92059589a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4290f95cc39945379be798778eae2e18" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42859 } } peers { permanent_uuid: "7fa1b50d6220465ab5e6409556e6ee7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36767 } } peers { permanent_uuid: "c6c5446de0824443bba78a92059589a9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45847 } }
I20260620 23:36:27.067950 7929 leader_election.cc:288] T 006d7fd72c7d44e7a9459887c0cf3771 P c6c5446de0824443bba78a92059589a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4290f95cc39945379be798778eae2e18 (127.0.97.2:42859), 7fa1b50d6220465ab5e6409556e6ee7e (127.0.97.3:36767)
I20260620 23:36:27.068465 7774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006d7fd72c7d44e7a9459887c0cf3771" candidate_uuid: "c6c5446de0824443bba78a92059589a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa1b50d6220465ab5e6409556e6ee7e" is_pre_election: true
I20260620 23:36:27.068468 7633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006d7fd72c7d44e7a9459887c0cf3771" candidate_uuid: "c6c5446de0824443bba78a92059589a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4290f95cc39945379be798778eae2e18" is_pre_election: true
I20260620 23:36:27.068720 7633 raft_consensus.cc:2454] T 006d7fd72c7d44e7a9459887c0cf3771 P 4290f95cc39945379be798778eae2e18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6c5446de0824443bba78a92059589a9 in term 0.
I20260620 23:36:27.068738 7774 raft_consensus.cc:2454] T 006d7fd72c7d44e7a9459887c0cf3771 P 7fa1b50d6220465ab5e6409556e6ee7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6c5446de0824443bba78a92059589a9 in term 0.
I20260620 23:36:27.069183 7421 leader_election.cc:302] T 006d7fd72c7d44e7a9459887c0cf3771 P c6c5446de0824443bba78a92059589a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4290f95cc39945379be798778eae2e18, c6c5446de0824443bba78a92059589a9; no voters:
I20260620 23:36:27.069490 7929 raft_consensus.cc:2781] T 006d7fd72c7d44e7a9459887c0cf3771 P c6c5446de0824443bba78a92059589a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:36:27.069609 7929 raft_consensus.cc:482] T 006d7fd72c7d44e7a9459887c0cf3771 P c6c5446de0824443bba78a92059589a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:36:27.069725 7929 raft_consensus.cc:3037] T 006d7fd72c7d44e7a9459887c0cf3771 P c6c5446de0824443bba78a92059589a9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:36:27.071859 7929 raft_consensus.cc:504] T 006d7fd72c7d44e7a9459887c0cf3771 P c6c5446de0824443bba78a92059589a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4290f95cc39945379be798778eae2e18" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42859 } } peers { permanent_uuid: "7fa1b50d6220465ab5e6409556e6ee7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36767 } } peers { permanent_uuid: "c6c5446de0824443bba78a92059589a9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45847 } }
I20260620 23:36:27.072507 7929 leader_election.cc:288] T 006d7fd72c7d44e7a9459887c0cf3771 P c6c5446de0824443bba78a92059589a9 [CANDIDATE]: Term 1 election: Requested vote from peers 4290f95cc39945379be798778eae2e18 (127.0.97.2:42859), 7fa1b50d6220465ab5e6409556e6ee7e (127.0.97.3:36767)
I20260620 23:36:27.072965 7633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006d7fd72c7d44e7a9459887c0cf3771" candidate_uuid: "c6c5446de0824443bba78a92059589a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4290f95cc39945379be798778eae2e18"
I20260620 23:36:27.073017 7774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006d7fd72c7d44e7a9459887c0cf3771" candidate_uuid: "c6c5446de0824443bba78a92059589a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa1b50d6220465ab5e6409556e6ee7e"
I20260620 23:36:27.073177 7633 raft_consensus.cc:3037] T 006d7fd72c7d44e7a9459887c0cf3771 P 4290f95cc39945379be798778eae2e18 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:36:27.073212 7774 raft_consensus.cc:3037] T 006d7fd72c7d44e7a9459887c0cf3771 P 7fa1b50d6220465ab5e6409556e6ee7e [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:36:27.075268 7633 raft_consensus.cc:2454] T 006d7fd72c7d44e7a9459887c0cf3771 P 4290f95cc39945379be798778eae2e18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6c5446de0824443bba78a92059589a9 in term 1.
I20260620 23:36:27.075747 7421 leader_election.cc:302] T 006d7fd72c7d44e7a9459887c0cf3771 P c6c5446de0824443bba78a92059589a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4290f95cc39945379be798778eae2e18, c6c5446de0824443bba78a92059589a9; no voters:
I20260620 23:36:27.076027 7929 raft_consensus.cc:2781] T 006d7fd72c7d44e7a9459887c0cf3771 P c6c5446de0824443bba78a92059589a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:36:27.076171 7774 raft_consensus.cc:2454] T 006d7fd72c7d44e7a9459887c0cf3771 P 7fa1b50d6220465ab5e6409556e6ee7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6c5446de0824443bba78a92059589a9 in term 1.
I20260620 23:36:27.076632 7929 raft_consensus.cc:686] T 006d7fd72c7d44e7a9459887c0cf3771 P c6c5446de0824443bba78a92059589a9 [term 1 LEADER]: Becoming Leader. State: Replica: c6c5446de0824443bba78a92059589a9, State: Running, Role: LEADER
I20260620 23:36:27.076938 7929 consensus_queue.cc:237] T 006d7fd72c7d44e7a9459887c0cf3771 P c6c5446de0824443bba78a92059589a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4290f95cc39945379be798778eae2e18" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42859 } } peers { permanent_uuid: "7fa1b50d6220465ab5e6409556e6ee7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36767 } } peers { permanent_uuid: "c6c5446de0824443bba78a92059589a9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45847 } }
I20260620 23:36:27.077732 7930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:27.077278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33681 (local address 127.0.97.1:58413)
0620 23:36:27.077439 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:27.077529 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 23:36:27.079888 7351 catalog_manager.cc:5495] T 006d7fd72c7d44e7a9459887c0cf3771 P c6c5446de0824443bba78a92059589a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6c5446de0824443bba78a92059589a9 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "c6c5446de0824443bba78a92059589a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4290f95cc39945379be798778eae2e18" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fa1b50d6220465ab5e6409556e6ee7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6c5446de0824443bba78a92059589a9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45847 } health_report { overall_health: HEALTHY } } }
I20260620 23:36:27.141829 7917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:27.136435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37292 (local address 127.0.97.1:45847)
0620 23:36:27.136658 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:27.136662 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:27.136680 (+ 18us) server_negotiation.cc:407] Connection header received
0620 23:36:27.136875 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:27.136879 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:27.136963 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:27.137048 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:36:27.137534 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:27.138399 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:27.139906 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:27.140461 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:27.140576 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:27.141029 (+ 453us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:36:27.141211 (+ 182us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:36:27.141317 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:27.141426 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 23:36:27.141493 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 23:36:27.145179 7633 raft_consensus.cc:1261] T 006d7fd72c7d44e7a9459887c0cf3771 P 4290f95cc39945379be798778eae2e18 [term 1 FOLLOWER]: Refusing update from remote peer c6c5446de0824443bba78a92059589a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:36:27.145205 7774 raft_consensus.cc:1261] T 006d7fd72c7d44e7a9459887c0cf3771 P 7fa1b50d6220465ab5e6409556e6ee7e [term 1 FOLLOWER]: Refusing update from remote peer c6c5446de0824443bba78a92059589a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:36:27.145917 7931 consensus_queue.cc:1035] T 006d7fd72c7d44e7a9459887c0cf3771 P c6c5446de0824443bba78a92059589a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4290f95cc39945379be798778eae2e18" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:36:27.146273 7929 consensus_queue.cc:1035] T 006d7fd72c7d44e7a9459887c0cf3771 P c6c5446de0824443bba78a92059589a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa1b50d6220465ab5e6409556e6ee7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:36:27.148958 7928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:27.148383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33681 (local address 127.0.97.3:41191)
0620 23:36:27.148610 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:27.148706 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260620 23:36:27.149335 7925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:27.148652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33681 (local address 127.0.97.2:46665)
0620 23:36:27.148873 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:27.149002 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 23:36:27.157153 7936 mvcc.cc:205] Tried to move back new op lower bound from 7299066212940247040 to 7299066212669751296. Current Snapshot: MvccSnapshot[applied={T|T < 7299066212940247040}]
I20260620 23:36:27.158571 7928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:27.158060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33681 (local address 127.0.97.3:35321)
0620 23:36:27.158283 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:27.158393 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260620 23:36:27.166115 7351 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:36:27.166507 7351 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:36:27.170158 7930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:27.165120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33681 (local address 127.0.97.1:59439)
0620 23:36:27.169872 (+ 4752us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:27.169989 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4623}
I20260620 23:36:27.174814 7925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:27.174366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33681 (local address 127.0.97.2:35813)
0620 23:36:27.174540 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:27.174639 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 23:36:27.175457 7930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:27.174168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33681 (local address 127.0.97.1:53665)
0620 23:36:27.175146 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:27.175246 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":865}
I20260620 23:36:27.179371 7351 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4b672bfe633a443591fc2dd3ca7277b1: Sending DeleteTablet for 3 replicas of tablet 006d7fd72c7d44e7a9459887c0cf3771
I20260620 23:36:27.180670 7613 tablet_service.cc:1514] Processing DeleteTablet for tablet 006d7fd72c7d44e7a9459887c0cf3771 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:36:27 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:47012
I20260620 23:36:27.180706 7754 tablet_service.cc:1514] Processing DeleteTablet for tablet 006d7fd72c7d44e7a9459887c0cf3771 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:36:27 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:48502
I20260620 23:36:27.180920 7753 tablet_service.cc:1514] Processing DeleteTablet for tablet 006d7fd72c7d44e7a9459887c0cf3771 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:36:27 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:48502
I20260620 23:36:27.181151 7612 tablet_service.cc:1514] Processing DeleteTablet for tablet 006d7fd72c7d44e7a9459887c0cf3771 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:36:27 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:47012
I20260620 23:36:27.182140 7465 tablet_service.cc:1514] Processing DeleteTablet for tablet 006d7fd72c7d44e7a9459887c0cf3771 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:36:27 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:37278
I20260620 23:36:27.182188 7464 tablet_service.cc:1514] Processing DeleteTablet for tablet 006d7fd72c7d44e7a9459887c0cf3771 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:36:27 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:37278
I20260620 23:36:27.182787 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 7331
I20260620 23:36:27.183032 7946 tablet_replica.cc:330] T 006d7fd72c7d44e7a9459887c0cf3771 P c6c5446de0824443bba78a92059589a9: stopping tablet replica
I20260620 23:36:27.183284 7946 raft_consensus.cc:2229] T 006d7fd72c7d44e7a9459887c0cf3771 P c6c5446de0824443bba78a92059589a9 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:36:27.183791 7946 raft_consensus.cc:2258] T 006d7fd72c7d44e7a9459887c0cf3771 P c6c5446de0824443bba78a92059589a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:36:27.186590 7946 ts_tablet_manager.cc:1905] T 006d7fd72c7d44e7a9459887c0cf3771 P c6c5446de0824443bba78a92059589a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:36:27.187881 7944 tablet_replica.cc:330] T 006d7fd72c7d44e7a9459887c0cf3771 P 4290f95cc39945379be798778eae2e18: stopping tablet replica
I20260620 23:36:27.188158 7944 raft_consensus.cc:2229] T 006d7fd72c7d44e7a9459887c0cf3771 P 4290f95cc39945379be798778eae2e18 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:36:27.188509 7944 raft_consensus.cc:2258] T 006d7fd72c7d44e7a9459887c0cf3771 P 4290f95cc39945379be798778eae2e18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:36:27.192534 7944 ts_tablet_manager.cc:1905] T 006d7fd72c7d44e7a9459887c0cf3771 P 4290f95cc39945379be798778eae2e18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:36:27.192718 7946 ts_tablet_manager.cc:1918] T 006d7fd72c7d44e7a9459887c0cf3771 P c6c5446de0824443bba78a92059589a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:36:27.192885 7946 log.cc:1196] T 006d7fd72c7d44e7a9459887c0cf3771 P c6c5446de0824443bba78a92059589a9: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-0/wal/wals/006d7fd72c7d44e7a9459887c0cf3771
I20260620 23:36:27.193331 7946 ts_tablet_manager.cc:1939] T 006d7fd72c7d44e7a9459887c0cf3771 P c6c5446de0824443bba78a92059589a9: Deleting consensus metadata
I20260620 23:36:27.194344 7336 catalog_manager.cc:4841] TS c6c5446de0824443bba78a92059589a9 (127.0.97.1:45847): tablet 006d7fd72c7d44e7a9459887c0cf3771 (table test-table [id=ed5035d48f8041a196c650ae242b32aa]) successfully deleted
W20260620 23:36:27.195113 7336 catalog_manager.cc:4808] TS c6c5446de0824443bba78a92059589a9 (127.0.97.1:45847): delete failed for tablet 006d7fd72c7d44e7a9459887c0cf3771 because the tablet was not found. No further retry: Not found: Tablet not found: 006d7fd72c7d44e7a9459887c0cf3771
I20260620 23:36:27.199390 7945 tablet_replica.cc:330] T 006d7fd72c7d44e7a9459887c0cf3771 P 7fa1b50d6220465ab5e6409556e6ee7e: stopping tablet replica
I20260620 23:36:27.199622 7945 raft_consensus.cc:2229] T 006d7fd72c7d44e7a9459887c0cf3771 P 7fa1b50d6220465ab5e6409556e6ee7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:36:27.199906 7945 raft_consensus.cc:2258] T 006d7fd72c7d44e7a9459887c0cf3771 P 7fa1b50d6220465ab5e6409556e6ee7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:36:27.200902 7945 ts_tablet_manager.cc:1905] T 006d7fd72c7d44e7a9459887c0cf3771 P 7fa1b50d6220465ab5e6409556e6ee7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:36:27.202411 7944 ts_tablet_manager.cc:1918] T 006d7fd72c7d44e7a9459887c0cf3771 P 4290f95cc39945379be798778eae2e18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:36:27.202560 7944 log.cc:1196] T 006d7fd72c7d44e7a9459887c0cf3771 P 4290f95cc39945379be798778eae2e18: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-1/wal/wals/006d7fd72c7d44e7a9459887c0cf3771
I20260620 23:36:27.203071 7944 ts_tablet_manager.cc:1939] T 006d7fd72c7d44e7a9459887c0cf3771 P 4290f95cc39945379be798778eae2e18: Deleting consensus metadata
I20260620 23:36:27.204412 7338 catalog_manager.cc:4841] TS 4290f95cc39945379be798778eae2e18 (127.0.97.2:42859): tablet 006d7fd72c7d44e7a9459887c0cf3771 (table test-table [id=ed5035d48f8041a196c650ae242b32aa]) successfully deleted
W20260620 23:36:27.204922 7338 catalog_manager.cc:4808] TS 4290f95cc39945379be798778eae2e18 (127.0.97.2:42859): delete failed for tablet 006d7fd72c7d44e7a9459887c0cf3771 because the tablet was not found. No further retry: Not found: Tablet not found: 006d7fd72c7d44e7a9459887c0cf3771
I20260620 23:36:27.206779 7945 ts_tablet_manager.cc:1918] T 006d7fd72c7d44e7a9459887c0cf3771 P 7fa1b50d6220465ab5e6409556e6ee7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:36:27.206960 7945 log.cc:1196] T 006d7fd72c7d44e7a9459887c0cf3771 P 7fa1b50d6220465ab5e6409556e6ee7e: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998525501163-388-0/minicluster-data/ts-2/wal/wals/006d7fd72c7d44e7a9459887c0cf3771
I20260620 23:36:27.207414 7945 ts_tablet_manager.cc:1939] T 006d7fd72c7d44e7a9459887c0cf3771 P 7fa1b50d6220465ab5e6409556e6ee7e: Deleting consensus metadata
I20260620 23:36:27.208379 7336 catalog_manager.cc:4841] TS 7fa1b50d6220465ab5e6409556e6ee7e (127.0.97.3:36767): tablet 006d7fd72c7d44e7a9459887c0cf3771 (table test-table [id=ed5035d48f8041a196c650ae242b32aa]) successfully deleted
W20260620 23:36:27.208673 7336 catalog_manager.cc:4808] TS 7fa1b50d6220465ab5e6409556e6ee7e (127.0.97.3:36767): delete failed for tablet 006d7fd72c7d44e7a9459887c0cf3771 because the tablet was not found. No further retry: Not found: Tablet not found: 006d7fd72c7d44e7a9459887c0cf3771
I20260620 23:36:27.211404 7917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:27.184234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37302 (local address 127.0.97.1:45847)
0620 23:36:27.184685 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:27.184690 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:27.187109 (+ 2419us) server_negotiation.cc:407] Connection header received
0620 23:36:27.187335 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:27.187343 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:27.187434 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:27.187539 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:27.191377 (+ 3838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:27.192465 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:27.200651 (+ 8186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:27.201292 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:27.201443 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:27.205068 (+ 3625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:27.205089 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:27.205109 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:27.205159 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:27.208898 (+ 3739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:27.209549 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:27.209555 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:27.209558 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:27.209698 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:27.210039 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:27.210043 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:27.210045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:27.210406 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:27.210623 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:27.210983 (+ 360us) server_negotiation.cc:299] Negotiation successful
0620 23:36:27.211166 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342}
I20260620 23:36:27.212045 7525 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:36:27.332464 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 7331
I20260620 23:36:27.355355 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 7542
I20260620 23:36:27.369294 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:27.356287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47028 (local address 127.0.97.2:42859)
0620 23:36:27.356497 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:27.356502 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:27.356519 (+ 17us) server_negotiation.cc:407] Connection header received
0620 23:36:27.356661 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:27.356665 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:27.356727 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:27.356812 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:27.358031 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:27.358754 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:27.360081 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:27.360481 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:27.360599 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:27.363190 (+ 2591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:27.363205 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:27.363222 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:27.363256 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:27.366524 (+ 3268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:27.367317 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:27.367325 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:27.367327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:27.367447 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:27.367779 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:27.367782 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:27.367784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:27.368132 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:27.368379 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:27.368742 (+ 363us) server_negotiation.cc:299] Negotiation successful
0620 23:36:27.368927 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 23:36:27.369990 7673 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:36:27.453066 7950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:27.452370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33681 (local address 127.0.0.1:50388)
0620 23:36:27.452808 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:27.452903 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":237,"threads_started":1}
I20260620 23:36:27.454380 7950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:27.454031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33681 (local address 127.0.0.1:50392)
0620 23:36:27.454175 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:27.454269 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 23:36:27.464870 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 7542
I20260620 23:36:27.483228 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 7691
I20260620 23:36:27.497800 7952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:27.484017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48518 (local address 127.0.97.3:36767)
0620 23:36:27.484503 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:27.484507 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:27.484534 (+ 27us) server_negotiation.cc:407] Connection header received
0620 23:36:27.484590 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:27.484593 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:27.484654 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:27.484749 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:27.486222 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:27.487080 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:27.488400 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:27.488814 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:27.488926 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:27.491739 (+ 2813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:27.491753 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:27.491769 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:27.491812 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:27.494753 (+ 2941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:27.495630 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:27.495638 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:27.495641 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:27.495776 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:27.496226 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:27.496229 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:27.496231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:27.496597 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:27.496877 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:27.497253 (+ 376us) server_negotiation.cc:299] Negotiation successful
0620 23:36:27.497439 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":277,"threads_started":1}
I20260620 23:36:27.498466 7814 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:36:27.599030 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 7691
I20260620 23:36:27.616571 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 7022
I20260620 23:36:27.617715 7190 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:36:27.735281 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 7022
I20260620 23:36:27.756198 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 7123
I20260620 23:36:27.757334 7381 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:36:27.879280 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 7123
2026-06-20T23:36:27Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9886 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:36:27 dist-test-slave-tcgx krb5kdc[7958](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:36:27 dist-test-slave-tcgx krb5kdc[7958](info): set up 2 sockets
Jun 20 23:36:27 dist-test-slave-tcgx krb5kdc[7958](info): commencing operation
krb5kdc: starting...
W20260620 23:36:29.036412 388 mini_kdc.cc:121] Time spent starting KDC: real 1.111s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/krb5kdc/test-user.keytab.
Jun 20 23:36:29 dist-test-slave-tcgx krb5kdc[7958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:36:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:36:29Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260620 23:36:29.263840 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34977
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34977
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:36:29.600288 7971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:36:29.600618 7971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:36:29.600711 7971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:36:29.608058 7971 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:36:29.608186 7971 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 23:36:29.608227 7971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:36:29.608280 7971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:36:29.608331 7971 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:36:29.608386 7971 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:36:29.618459 7971 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34977
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:36:29.619798 7971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:36:29.621093 7971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:36:29.633214 7979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:29.633276 7976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:29.633251 7977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:36:29.634095 7971 server_base.cc:1034] running on GCE node
I20260620 23:36:29.635021 7971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:36:29.636533 7971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:36:29.637818 7971 hybrid_clock.cc:648] HybridClock initialized: now 1781998589637716 us; error 63 us; skew 500 ppm
I20260620 23:36:29.638188 7971 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:36:29 dist-test-slave-tcgx krb5kdc[7958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998589, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:36:29.641530 7971 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260620 23:36:29.643296 7971 webserver.cc:460] Webserver started at http://127.0.97.62:42529/ using document root <none> and password file <none>
I20260620 23:36:29.644083 7971 fs_manager.cc:359] Metadata directory not provided
I20260620 23:36:29.644218 7971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:36:29.644577 7971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:36:29.647720 7971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/master-0/data/instance:
uuid: "09f79b3ac58944c1be69f25d4e293a77"
format_stamp: "Formatted at 2026-06-20 23:36:29 on dist-test-slave-tcgx"
server_key: "404fcfd2b41d12fefe8331220226788e"
server_key_iv: "336c27d103e1434dd9122936df127243"
server_key_version: "encryptionkey@0"
I20260620 23:36:29.648610 7971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/master-0/wal/instance:
uuid: "09f79b3ac58944c1be69f25d4e293a77"
format_stamp: "Formatted at 2026-06-20 23:36:29 on dist-test-slave-tcgx"
server_key: "404fcfd2b41d12fefe8331220226788e"
server_key_iv: "336c27d103e1434dd9122936df127243"
server_key_version: "encryptionkey@0"
I20260620 23:36:29.654027 7971 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 23:36:29.657801 7986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:29.659368 7971 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:36:29.659574 7971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/master-0/wal
uuid: "09f79b3ac58944c1be69f25d4e293a77"
format_stamp: "Formatted at 2026-06-20 23:36:29 on dist-test-slave-tcgx"
server_key: "404fcfd2b41d12fefe8331220226788e"
server_key_iv: "336c27d103e1434dd9122936df127243"
server_key_version: "encryptionkey@0"
I20260620 23:36:29.659818 7971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:36:29.679700 7971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:36:29.684356 7971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:36:29.684686 7971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:36:29.706871 7971 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34977
I20260620 23:36:29.706881 8047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34977 every 8 connection(s)
I20260620 23:36:29.708300 7971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:36:29.712105 8048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:36:29.716241 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 7971
I20260620 23:36:29.716487 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/master-0/wal/instance
I20260620 23:36:29.717309 388 external_mini_cluster.cc:1428] Setting key 6a65e5f89e3738d4d4a91b08280c52a4
I20260620 23:36:29.724391 8048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09f79b3ac58944c1be69f25d4e293a77: Bootstrap starting.
I20260620 23:36:29.727843 8048 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09f79b3ac58944c1be69f25d4e293a77: Neither blocks nor log segments found. Creating new log.
I20260620 23:36:29.729300 8048 log.cc:824] T 00000000000000000000000000000000 P 09f79b3ac58944c1be69f25d4e293a77: Log is configured to *not* fsync() on all Append() calls
I20260620 23:36:29.732681 8048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09f79b3ac58944c1be69f25d4e293a77: No bootstrap required, opened a new log
Jun 20 23:36:29 dist-test-slave-tcgx krb5kdc[7958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 23:36:29.740077 8048 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09f79b3ac58944c1be69f25d4e293a77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f79b3ac58944c1be69f25d4e293a77" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34977 } }
I20260620 23:36:29.740494 8048 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09f79b3ac58944c1be69f25d4e293a77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:36:29.740625 8048 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09f79b3ac58944c1be69f25d4e293a77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09f79b3ac58944c1be69f25d4e293a77, State: Initialized, Role: FOLLOWER
I20260620 23:36:29.741273 8048 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09f79b3ac58944c1be69f25d4e293a77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f79b3ac58944c1be69f25d4e293a77" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34977 } }
I20260620 23:36:29.741528 8048 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09f79b3ac58944c1be69f25d4e293a77 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:36:29.741677 8048 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09f79b3ac58944c1be69f25d4e293a77 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:36:29.741871 8048 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09f79b3ac58944c1be69f25d4e293a77 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:36:29.744350 8048 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09f79b3ac58944c1be69f25d4e293a77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f79b3ac58944c1be69f25d4e293a77" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34977 } }
I20260620 23:36:29.744956 8048 leader_election.cc:302] T 00000000000000000000000000000000 P 09f79b3ac58944c1be69f25d4e293a77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09f79b3ac58944c1be69f25d4e293a77; no voters:
I20260620 23:36:29.745640 8048 leader_election.cc:288] T 00000000000000000000000000000000 P 09f79b3ac58944c1be69f25d4e293a77 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:36:29.745803 8053 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09f79b3ac58944c1be69f25d4e293a77 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:36:29.747560 8051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:29.719674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40850 (local address 127.0.97.62:34977)
0620 23:36:29.720510 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:29.720524 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:29.720568 (+ 44us) server_negotiation.cc:407] Connection header received
0620 23:36:29.722919 (+ 2351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:29.723009 (+ 90us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:29.723498 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:29.723924 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:29.725467 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:29.726804 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:29.728347 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:29.729470 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:29.729660 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:29.738396 (+ 8736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:29.738405 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:29.738424 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:29.738476 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:29.742157 (+ 3681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:29.742919 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:29.742960 (+ 41us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:29.742971 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:29.743176 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:29.743629 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:29.743634 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:29.743637 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:29.744220 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:29.744575 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:29.745066 (+ 491us) server_negotiation.cc:299] Negotiation successful
0620 23:36:29.745655 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":377,"threads_started":1}
I20260620 23:36:29.748039 8048 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09f79b3ac58944c1be69f25d4e293a77 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:36:29.752552 8053 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09f79b3ac58944c1be69f25d4e293a77 [term 1 LEADER]: Becoming Leader. State: Replica: 09f79b3ac58944c1be69f25d4e293a77, State: Running, Role: LEADER
I20260620 23:36:29.753454 8053 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09f79b3ac58944c1be69f25d4e293a77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f79b3ac58944c1be69f25d4e293a77" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34977 } }
I20260620 23:36:29.756592 8054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09f79b3ac58944c1be69f25d4e293a77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09f79b3ac58944c1be69f25d4e293a77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f79b3ac58944c1be69f25d4e293a77" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34977 } } }
I20260620 23:36:29.756702 8055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09f79b3ac58944c1be69f25d4e293a77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09f79b3ac58944c1be69f25d4e293a77. Latest consensus state: current_term: 1 leader_uuid: "09f79b3ac58944c1be69f25d4e293a77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f79b3ac58944c1be69f25d4e293a77" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34977 } } }
I20260620 23:36:29.757037 8054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09f79b3ac58944c1be69f25d4e293a77 [sys.catalog]: This master's current role is: LEADER
I20260620 23:36:29.757104 8055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09f79b3ac58944c1be69f25d4e293a77 [sys.catalog]: This master's current role is: LEADER
W20260620 23:36:29.763650 8067 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 09f79b3ac58944c1be69f25d4e293a77: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:36:29.763820 8067 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:36:29.763960 8068 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:36:29.765785 8068 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:36:29.774127 8068 catalog_manager.cc:1347] Generated new cluster ID: 08c8f40a2d0a4530b94802ac8396aeb9
I20260620 23:36:29.774266 8068 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:36:29.799108 8068 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:36:29.800640 8068 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:36:29.808223 8068 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09f79b3ac58944c1be69f25d4e293a77: Generated new TSK 0
I20260620 23:36:29.809132 8068 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260620 23:36:29.871461 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34977
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:36:30.214699 8074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:36:30.215029 8074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:36:30.215129 8074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:36:30.222697 8074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:36:30.222821 8074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:36:30.222908 8074 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 23:36:30.223060 8074 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 23:36:30.233479 8074 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:34977
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:36:30.234798 8074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:36:30.236192 8074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:36:30.249809 8082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:30.249931 8079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:30.250257 8080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:36:30.251652 8074 server_base.cc:1034] running on GCE node
I20260620 23:36:30.252451 8074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:36:30.253706 8074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:36:30.255095 8074 hybrid_clock.cc:648] HybridClock initialized: now 1781998590255023 us; error 40 us; skew 500 ppm
I20260620 23:36:30.255489 8074 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:36:30 dist-test-slave-tcgx krb5kdc[7958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998590, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:36:30.259390 8074 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260620 23:36:30.261000 8074 webserver.cc:460] Webserver started at http://127.0.97.1:44981/ using document root <none> and password file <none>
I20260620 23:36:30.261765 8074 fs_manager.cc:359] Metadata directory not provided
I20260620 23:36:30.261904 8074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:36:30.262250 8074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:36:30.265481 8074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/ts-0/data/instance:
uuid: "3515e5dbdb114e0dbe8c96b2af448ce3"
format_stamp: "Formatted at 2026-06-20 23:36:30 on dist-test-slave-tcgx"
server_key: "e8f3c8eea45157265c96b4909dd4be2e"
server_key_iv: "64ce835691366f51115bc385160db91e"
server_key_version: "encryptionkey@0"
I20260620 23:36:30.266309 8074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/ts-0/wal/instance:
uuid: "3515e5dbdb114e0dbe8c96b2af448ce3"
format_stamp: "Formatted at 2026-06-20 23:36:30 on dist-test-slave-tcgx"
server_key: "e8f3c8eea45157265c96b4909dd4be2e"
server_key_iv: "64ce835691366f51115bc385160db91e"
server_key_version: "encryptionkey@0"
I20260620 23:36:30.271777 8074 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 23:36:30.275393 8089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:30.276803 8074 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 23:36:30.276997 8074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/ts-0/wal
uuid: "3515e5dbdb114e0dbe8c96b2af448ce3"
format_stamp: "Formatted at 2026-06-20 23:36:30 on dist-test-slave-tcgx"
server_key: "e8f3c8eea45157265c96b4909dd4be2e"
server_key_iv: "64ce835691366f51115bc385160db91e"
server_key_version: "encryptionkey@0"
I20260620 23:36:30.277241 8074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:36:30.292645 8074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:36:30.297052 8074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:36:30.297356 8074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:36:30.300172 8074 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:36:30.300297 8074 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:30.300486 8074 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:36:30.300594 8074 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 23:36:30 dist-test-slave-tcgx krb5kdc[7958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998590, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 23:36:30.327852 8098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:30.300872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42293 (local address 127.0.97.62:34977)
0620 23:36:30.301444 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:30.301450 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:30.302606 (+ 1156us) server_negotiation.cc:407] Connection header received
0620 23:36:30.305108 (+ 2502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:30.305114 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:30.305204 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:30.305287 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:30.308061 (+ 2774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:30.309191 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:30.311849 (+ 2658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:30.312633 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:30.312814 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:30.318691 (+ 5877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:30.318698 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:30.318704 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:30.318754 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:30.323008 (+ 4254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:30.324149 (+ 1141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:30.324159 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:30.324166 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:30.324348 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:30.325082 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:30.325088 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:30.325091 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:30.325551 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:30.325726 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:30.326986 (+ 1260us) server_negotiation.cc:299] Negotiation successful
0620 23:36:30.327312 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":326,"threads_started":1}
I20260620 23:36:30.329797 8099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:30.301564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34977 (local address 127.0.97.1:42293)
0620 23:36:30.302415 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:30.302461 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:30.304697 (+ 2236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:30.305887 (+ 1190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:30.305896 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:30.306774 (+ 878us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:30.307855 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:30.307879 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:30.309404 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:30.309408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:30.311596 (+ 2188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:30.311620 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:30.312924 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:30.312932 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:30.313198 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:30.314086 (+ 888us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:30.314141 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:30.318289 (+ 4148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:36:30.323297 (+ 5008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:30.323316 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:30.323344 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:30.323975 (+ 631us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:30.324514 (+ 539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:30.324523 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:30.324529 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:30.324917 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:30.325709 (+ 792us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:36:30.325722 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:36:30.326412 (+ 690us) client_negotiation.cc:769] Sending connection context
0620 23:36:30.326825 (+ 413us) client_negotiation.cc:240] Negotiation successful
0620 23:36:30.327610 (+ 785us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":343,"threads_started":1}
I20260620 23:36:30.345017 8074 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40283
I20260620 23:36:30.345018 8209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40283 every 8 connection(s)
I20260620 23:36:30.346524 8074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
I20260620 23:36:30.348729 8210 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34977
I20260620 23:36:30.348978 8210 heartbeater.cc:464] Registering TS with master...
I20260620 23:36:30.349787 8210 heartbeater.cc:511] Master 127.0.97.62:34977 requested a full tablet report, sending...
I20260620 23:36:30.352212 8003 ts_manager.cc:194] Registered new tserver with Master: 3515e5dbdb114e0dbe8c96b2af448ce3 (127.0.97.1:40283)
I20260620 23:36:30.353343 8098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:30.337568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43265 (local address 127.0.97.62:34977)
0620 23:36:30.337809 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:30.337818 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:30.339107 (+ 1289us) server_negotiation.cc:407] Connection header received
0620 23:36:30.339344 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:30.339351 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:30.339451 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:30.339557 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:30.341189 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:30.342253 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:30.344013 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:30.344786 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:30.344933 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:30.347116 (+ 2183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:30.347122 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:30.347126 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:30.347165 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:30.350336 (+ 3171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:30.351187 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:30.351192 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:30.351196 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:30.351358 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:30.351706 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:30.351712 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:30.351716 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:30.352110 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:30.352267 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:30.352721 (+ 454us) server_negotiation.cc:299] Negotiation successful
0620 23:36:30.352983 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 23:36:30.353377 8192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:30.337991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34977 (local address 127.0.97.1:43265)
0620 23:36:30.338970 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:30.338996 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:30.339184 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:30.339760 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:30.339765 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:30.340489 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:30.341010 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:30.341034 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:30.342372 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:30.342381 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:30.343820 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:30.343840 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:30.345025 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:30.345034 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:30.345259 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:30.345984 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:30.346001 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:30.346857 (+ 856us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:36:30.350464 (+ 3607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:30.350469 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:30.350473 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:30.351060 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:30.351468 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:30.351471 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:30.351473 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:30.351589 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:30.352242 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:36:30.352246 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:36:30.352444 (+ 198us) client_negotiation.cc:769] Sending connection context
0620 23:36:30.352589 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 23:36:30.353033 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":806,"thread_start_us":658,"threads_started":1}
I20260620 23:36:30.354151 8003 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:42293
I20260620 23:36:30.354517 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 8074
I20260620 23:36:30.354743 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/ts-0/wal/instance
I20260620 23:36:30.355571 388 external_mini_cluster.cc:1428] Setting key c2d9e2c48e7b7d0c76bc9ebab7fe9404
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260620 23:36:30.406806 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34977
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:36:30.741184 8214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:36:30.741536 8214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:36:30.741716 8214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:36:30.749418 8214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:36:30.749589 8214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:36:30.749737 8214 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 23:36:30.749871 8214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 23:36:30.760254 8214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:34977
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:36:30.761682 8214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:36:30.763120 8214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:36:30.777158 8220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:30.777230 8219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:30.777186 8222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:36:30.778769 8214 server_base.cc:1034] running on GCE node
I20260620 23:36:30.779481 8214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:36:30.780562 8214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:36:30.781817 8214 hybrid_clock.cc:648] HybridClock initialized: now 1781998590781733 us; error 53 us; skew 500 ppm
I20260620 23:36:30.782171 8214 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:36:30 dist-test-slave-tcgx krb5kdc[7958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998590, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:36:30.785526 8214 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260620 23:36:30.787288 8214 webserver.cc:460] Webserver started at http://127.0.97.2:42023/ using document root <none> and password file <none>
I20260620 23:36:30.788054 8214 fs_manager.cc:359] Metadata directory not provided
I20260620 23:36:30.788146 8214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:36:30.788523 8214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:36:30.791759 8214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/ts-1/data/instance:
uuid: "e347312aaa774f81b9f0232b2f7f1d8d"
format_stamp: "Formatted at 2026-06-20 23:36:30 on dist-test-slave-tcgx"
server_key: "719bd73dc02f260a9aa639ac7ef503a9"
server_key_iv: "6df66156e8a3f6b54659e1e89702d946"
server_key_version: "encryptionkey@0"
I20260620 23:36:30.792614 8214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/ts-1/wal/instance:
uuid: "e347312aaa774f81b9f0232b2f7f1d8d"
format_stamp: "Formatted at 2026-06-20 23:36:30 on dist-test-slave-tcgx"
server_key: "719bd73dc02f260a9aa639ac7ef503a9"
server_key_iv: "6df66156e8a3f6b54659e1e89702d946"
server_key_version: "encryptionkey@0"
I20260620 23:36:30.798105 8214 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 23:36:30.801813 8229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:30.803344 8214 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 23:36:30.803534 8214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/ts-1/wal
uuid: "e347312aaa774f81b9f0232b2f7f1d8d"
format_stamp: "Formatted at 2026-06-20 23:36:30 on dist-test-slave-tcgx"
server_key: "719bd73dc02f260a9aa639ac7ef503a9"
server_key_iv: "6df66156e8a3f6b54659e1e89702d946"
server_key_version: "encryptionkey@0"
I20260620 23:36:30.803798 8214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:36:30.851704 8214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:36:30.856434 8214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:36:30.856724 8214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:36:30.859676 8214 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:36:30.859797 8214 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:30.859980 8214 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:36:30.860056 8214 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 23:36:30 dist-test-slave-tcgx krb5kdc[7958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998590, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 23:36:30.885989 8238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:30.860300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59029 (local address 127.0.97.62:34977)
0620 23:36:30.860975 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:30.860981 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:30.862207 (+ 1226us) server_negotiation.cc:407] Connection header received
0620 23:36:30.865387 (+ 3180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:30.865397 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:30.865518 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:30.865620 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:30.868815 (+ 3195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:30.869603 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:30.871740 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:30.872341 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:30.872497 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:30.878506 (+ 6009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:30.878511 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:30.878516 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:30.878553 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:30.881927 (+ 3374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:30.883125 (+ 1198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:30.883133 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:30.883138 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:30.883295 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:30.883918 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:30.883923 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:30.883927 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:30.884339 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:30.884495 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:30.885364 (+ 869us) server_negotiation.cc:299] Negotiation successful
0620 23:36:30.885585 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":445,"threads_started":1}
I20260620 23:36:30.887274 8239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:30.861123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34977 (local address 127.0.97.2:59029)
0620 23:36:30.862009 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:30.862064 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:30.864944 (+ 2880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:30.866329 (+ 1385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:30.866346 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:30.867352 (+ 1006us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:30.868496 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:30.868561 (+ 65us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:30.869850 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:30.869854 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:30.871551 (+ 1697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:30.871569 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:30.872641 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:30.872651 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:30.873093 (+ 442us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:30.874008 (+ 915us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:30.874053 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:30.878162 (+ 4109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:36:30.882163 (+ 4001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:30.882174 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:30.882204 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:30.882890 (+ 686us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:30.883420 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:30.883425 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:30.883429 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:30.883783 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:30.884569 (+ 786us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:36:30.884587 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:36:30.884984 (+ 397us) client_negotiation.cc:769] Sending connection context
0620 23:36:30.885268 (+ 284us) client_negotiation.cc:240] Negotiation successful
0620 23:36:30.885838 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":299,"threads_started":1}
I20260620 23:36:30.910382 8238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:30.894649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53427 (local address 127.0.97.62:34977)
0620 23:36:30.894886 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:30.894895 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:30.895747 (+ 852us) server_negotiation.cc:407] Connection header received
0620 23:36:30.895997 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:30.896004 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:30.896097 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:30.896230 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:30.897745 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:30.898661 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:30.900940 (+ 2279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:30.901544 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:30.901684 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:30.903838 (+ 2154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:30.903843 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:30.903847 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:30.903877 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:30.906495 (+ 2618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:30.907829 (+ 1334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:30.907836 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:30.907841 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:30.907995 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:30.908538 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:30.908544 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:30.908548 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:30.908991 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:30.909176 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:30.909717 (+ 541us) server_negotiation.cc:299] Negotiation successful
0620 23:36:30.909970 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 23:36:30.910466 8297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:30.894619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34977 (local address 127.0.97.2:53427)
0620 23:36:30.895594 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:30.895623 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:30.895823 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:30.896259 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:30.896264 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:30.897058 (+ 794us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:30.897584 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:30.897596 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:30.898820 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:30.898826 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:30.900721 (+ 1895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:30.900744 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:30.901753 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:30.901762 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:30.901920 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:30.902642 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:30.902711 (+ 69us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:30.903614 (+ 903us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:36:30.906697 (+ 3083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:30.906702 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:30.906706 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:30.907513 (+ 807us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:30.908152 (+ 639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:30.908161 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:30.908168 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:30.908390 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:30.909123 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:36:30.909128 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:36:30.909405 (+ 277us) client_negotiation.cc:769] Sending connection context
0620 23:36:30.909593 (+ 188us) client_negotiation.cc:240] Negotiation successful
0620 23:36:30.909970 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":748,"thread_start_us":565,"threads_started":1}
I20260620 23:36:30.920152 8214 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44049
I20260620 23:36:30.920207 8349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44049 every 8 connection(s)
I20260620 23:36:30.921468 8214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
I20260620 23:36:30.923936 8350 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34977
I20260620 23:36:30.924180 8350 heartbeater.cc:464] Registering TS with master...
I20260620 23:36:30.924887 8350 heartbeater.cc:511] Master 127.0.97.62:34977 requested a full tablet report, sending...
I20260620 23:36:30.926290 8003 ts_manager.cc:194] Registered new tserver with Master: e347312aaa774f81b9f0232b2f7f1d8d (127.0.97.2:44049)
I20260620 23:36:30.927205 8003 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:59029
I20260620 23:36:30.929929 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 8214
I20260620 23:36:30.930120 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/ts-1/wal/instance
I20260620 23:36:30.930541 388 external_mini_cluster.cc:1428] Setting key 5bb1fd17ea050c20b08c138654df2983
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260620 23:36:30.977797 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34977
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:36:31.316550 8354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:36:31.316922 8354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:36:31.317102 8354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:36:31.324680 8354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:36:31.324852 8354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:36:31.325003 8354 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 23:36:31.325139 8354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 23:36:31.335460 8354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:34977
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:36:31.336859 8354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:36:31.338274 8354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:36:31.352717 8359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:31.352669 8360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:31.352648 8362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:36:31.354079 8354 server_base.cc:1034] running on GCE node
I20260620 23:36:31.354859 8354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:36:31.355852 8354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:36:31.357079 8354 hybrid_clock.cc:648] HybridClock initialized: now 1781998591357002 us; error 52 us; skew 500 ppm
I20260620 23:36:31.357367 8354 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:36:31.358203 8210 heartbeater.cc:503] Master 127.0.97.62:34977 was elected leader, sending a full tablet report...
Jun 20 23:36:31 dist-test-slave-tcgx krb5kdc[7958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998591, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:36:31.361029 8354 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260620 23:36:31.362504 8354 webserver.cc:460] Webserver started at http://127.0.97.3:39901/ using document root <none> and password file <none>
I20260620 23:36:31.363334 8354 fs_manager.cc:359] Metadata directory not provided
I20260620 23:36:31.363444 8354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:36:31.363744 8354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:36:31.367029 8354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/ts-2/data/instance:
uuid: "86d9648f36394a698d81a8dc587f77c1"
format_stamp: "Formatted at 2026-06-20 23:36:31 on dist-test-slave-tcgx"
server_key: "d361ddbd881fdac2ad6c98cfdf1acf25"
server_key_iv: "303176df348f93e124ee3f74acbe47b8"
server_key_version: "encryptionkey@0"
I20260620 23:36:31.367812 8354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/ts-2/wal/instance:
uuid: "86d9648f36394a698d81a8dc587f77c1"
format_stamp: "Formatted at 2026-06-20 23:36:31 on dist-test-slave-tcgx"
server_key: "d361ddbd881fdac2ad6c98cfdf1acf25"
server_key_iv: "303176df348f93e124ee3f74acbe47b8"
server_key_version: "encryptionkey@0"
I20260620 23:36:31.373054 8354 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:36:31.376854 8369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:31.378322 8354 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 23:36:31.378469 8354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/ts-2/wal
uuid: "86d9648f36394a698d81a8dc587f77c1"
format_stamp: "Formatted at 2026-06-20 23:36:31 on dist-test-slave-tcgx"
server_key: "d361ddbd881fdac2ad6c98cfdf1acf25"
server_key_iv: "303176df348f93e124ee3f74acbe47b8"
server_key_version: "encryptionkey@0"
I20260620 23:36:31.378664 8354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:36:31.401845 8354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:36:31.406127 8354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:36:31.406358 8354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:36:31.409233 8354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:36:31.409320 8354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:31.409521 8354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:36:31.409619 8354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 23:36:31 dist-test-slave-tcgx krb5kdc[7958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998591, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 23:36:31.436599 8238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:31.410354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58113 (local address 127.0.97.62:34977)
0620 23:36:31.410577 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:31.410585 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:31.412149 (+ 1564us) server_negotiation.cc:407] Connection header received
0620 23:36:31.414280 (+ 2131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:31.414286 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:31.414385 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:31.414494 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:31.417010 (+ 2516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:31.418105 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:31.420832 (+ 2727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:31.421223 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:31.421400 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:31.427858 (+ 6458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:31.427866 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:31.427871 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:31.427920 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:31.432240 (+ 4320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:31.433663 (+ 1423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:31.433672 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:31.433676 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:31.433849 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:31.434535 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:31.434541 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:31.434547 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:31.435021 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:31.435189 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:31.435973 (+ 784us) server_negotiation.cc:299] Negotiation successful
0620 23:36:31.436225 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260620 23:36:31.437672 8378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:31.411153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34977 (local address 127.0.97.3:58113)
0620 23:36:31.411972 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:31.412016 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:31.413956 (+ 1940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:31.414830 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:31.414850 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:31.415794 (+ 944us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:31.416791 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:31.416825 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:31.418309 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:31.418314 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:31.420593 (+ 2279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:31.420617 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:31.421557 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:31.421568 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:31.421992 (+ 424us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:31.422909 (+ 917us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:31.422977 (+ 68us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:31.427486 (+ 4509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:36:31.432482 (+ 4996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:31.432500 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:31.432529 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:31.433439 (+ 910us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:31.434028 (+ 589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:31.434038 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:31.434042 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:31.434359 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:31.435163 (+ 804us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:36:31.435172 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:36:31.435578 (+ 406us) client_negotiation.cc:769] Sending connection context
0620 23:36:31.435861 (+ 283us) client_negotiation.cc:240] Negotiation successful
0620 23:36:31.436430 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":313,"threads_started":1}
I20260620 23:36:31.459622 8354 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40371
I20260620 23:36:31.459630 8488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40371 every 8 connection(s)
I20260620 23:36:31.460916 8238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:31.445597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46401 (local address 127.0.97.62:34977)
0620 23:36:31.445802 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:31.445810 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:31.446615 (+ 805us) server_negotiation.cc:407] Connection header received
0620 23:36:31.446831 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:31.446839 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:31.446990 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:31.447090 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:31.448286 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:31.449177 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:31.450489 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:31.450926 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:31.451121 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:31.454051 (+ 2930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:31.454060 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:31.454068 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:31.454108 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:31.457934 (+ 3826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:31.458829 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:31.458834 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:31.458845 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:31.459042 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:31.459445 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:31.459451 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:31.459455 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:31.459766 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:31.459916 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:31.460331 (+ 415us) server_negotiation.cc:299] Negotiation successful
0620 23:36:31.460554 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 23:36:31.461053 8462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:31.445651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34977 (local address 127.0.97.3:46401)
0620 23:36:31.446483 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:31.446500 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:31.446675 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:31.447244 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:31.447249 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:31.447813 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:31.448132 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:31.448141 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:31.449308 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:31.449313 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:31.450343 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:31.450354 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:31.451201 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:31.451209 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:31.451512 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:31.452363 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:31.452397 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:31.453770 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:36:31.458101 (+ 4331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:31.458106 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:31.458111 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:31.458646 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:31.459180 (+ 534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:31.459190 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:31.459196 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:31.459324 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:31.459894 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:36:31.459902 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:36:31.460083 (+ 181us) client_negotiation.cc:769] Sending connection context
0620 23:36:31.460226 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 23:36:31.460590 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":707,"thread_start_us":579,"threads_started":1}
I20260620 23:36:31.461216 8354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
I20260620 23:36:31.462110 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 8354
I20260620 23:36:31.462360 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/ts-2/wal/instance
I20260620 23:36:31.463054 388 external_mini_cluster.cc:1428] Setting key f94bf797a235f0e88746b2e5f530e50f
I20260620 23:36:31.463308 8489 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34977
I20260620 23:36:31.463552 8489 heartbeater.cc:464] Registering TS with master...
I20260620 23:36:31.464461 8489 heartbeater.cc:511] Master 127.0.97.62:34977 requested a full tablet report, sending...
I20260620 23:36:31.466586 8002 ts_manager.cc:194] Registered new tserver with Master: 86d9648f36394a698d81a8dc587f77c1 (127.0.97.3:40371)
I20260620 23:36:31.467592 8002 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:58113
I20260620 23:36:31.479591 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 23:36:31 dist-test-slave-tcgx krb5kdc[7958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998591, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 23:36:31 dist-test-slave-tcgx krb5kdc[7958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:36:31 dist-test-slave-tcgx krb5kdc[7958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 23:36:31.535007 8238 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:36:31.523677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40856 (local address 127.0.97.62:34977)
0620 23:36:31.523941 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:31.523950 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:31.524204 (+ 254us) server_negotiation.cc:407] Connection header received
0620 23:36:31.524439 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:31.524445 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:31.524550 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:31.524662 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:31.526271 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:31.527126 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:31.528385 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:31.528769 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:31.528870 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:31.534670 (+ 5800us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40856: BlockingRecv error: recv got EOF from 127.0.0.1:40856 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Jun 20 23:36:31 dist-test-slave-tcgx krb5kdc[7958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998591, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 23:36:31.552398 8238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:31.538239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40872 (local address 127.0.97.62:34977)
0620 23:36:31.538481 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:31.538489 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:31.538721 (+ 232us) server_negotiation.cc:407] Connection header received
0620 23:36:31.538982 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:31.538988 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:31.539066 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:31.539173 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:31.540470 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:31.541334 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:31.542611 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:31.543109 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:31.543251 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:31.547190 (+ 3939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:31.547195 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:31.547199 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:31.547236 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:31.549895 (+ 2659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:31.550595 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:31.550599 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:31.550602 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:31.550738 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:31.551233 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:31.551237 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:31.551239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:31.551502 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:31.551702 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:31.551999 (+ 297us) server_negotiation.cc:299] Negotiation successful
0620 23:36:31.552157 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 23:36:31.556718 8002 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40872:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:36:31.563694 8002 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:36:31.586385 8507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:31.575965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40283 (local address 127.0.0.1:41472)
0620 23:36:31.576843 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:31.576886 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:31.577077 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:31.578153 (+ 1076us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:31.578168 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:31.578204 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:36:31.578682 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:31.578697 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:31.581292 (+ 2595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:31.581300 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:31.583627 (+ 2327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:31.583653 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:31.585628 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:31.585647 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:31.585850 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:31.585891 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 23:36:31.585980 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 23:36:31.586118 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":420,"threads_started":1}
I20260620 23:36:31.587538 8508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:31.576165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41472 (local address 127.0.97.1:40283)
0620 23:36:31.577430 (+ 1265us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:31.577439 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:31.577468 (+ 29us) server_negotiation.cc:407] Connection header received
0620 23:36:31.577616 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:31.577626 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:31.577902 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:31.578900 (+ 998us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:36:31.579751 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:31.581121 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:31.583848 (+ 2727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:31.585408 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:31.586715 (+ 1307us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:31.586980 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:31.587068 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 23:36:31.587248 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1107,"thread_start_us":966,"threads_started":1}
I20260620 23:36:31.587769 8511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:31.578005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44049 (local address 127.0.0.1:42826)
0620 23:36:31.578505 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:31.578521 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:31.578646 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:31.580158 (+ 1512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:31.580163 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:31.580192 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:36:31.580639 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:31.580656 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:31.582716 (+ 2060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:31.582722 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:31.585172 (+ 2450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:31.585188 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:31.586865 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:31.586871 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:31.587064 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:31.587103 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 23:36:31.587198 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 23:36:31.587356 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"mutex_wait_us":127,"thread_start_us":744,"threads_started":1}
I20260620 23:36:31.588249 8509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:31.577343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40371 (local address 127.0.0.1:36820)
0620 23:36:31.577900 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:31.577918 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:31.578032 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:31.579921 (+ 1889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:31.579926 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:31.579953 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:36:31.580389 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:31.580403 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:31.583023 (+ 2620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:31.583028 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:31.584953 (+ 1925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:31.584965 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:31.587151 (+ 2186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:31.587157 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:31.587356 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:31.587397 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 23:36:31.587459 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 23:36:31.587599 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"mutex_wait_us":55,"thread_start_us":307,"threads_started":1}
I20260620 23:36:31.588717 8510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:31.577747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36820 (local address 127.0.97.3:40371)
0620 23:36:31.579273 (+ 1526us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:31.579282 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:31.579316 (+ 34us) server_negotiation.cc:407] Connection header received
0620 23:36:31.579413 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:31.579420 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:31.579699 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:31.580538 (+ 839us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:36:31.580888 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:31.582623 (+ 1735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:31.585104 (+ 2481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:31.586897 (+ 1793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:31.587860 (+ 963us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:31.588091 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:31.588215 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 23:36:31.588397 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1359,"thread_start_us":1169,"threads_started":1}
I20260620 23:36:31.588800 8512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:31.578532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42826 (local address 127.0.97.2:44049)
0620 23:36:31.579549 (+ 1017us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:31.579583 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:31.579613 (+ 30us) server_negotiation.cc:407] Connection header received
0620 23:36:31.579714 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:31.579721 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:31.579938 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:31.580850 (+ 912us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:36:31.581047 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:31.582503 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:31.585336 (+ 2833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:31.586634 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:31.586819 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:31.588272 (+ 1453us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:31.588353 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 23:36:31.588516 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":880,"thread_start_us":748,"threads_started":1}
I20260620 23:36:31.591708 8140 tablet_service.cc:1467] Processing CreateTablet for tablet 11c314b7fcb545068f6f73dce1bd1cab (DEFAULT_TABLE table=test-table [id=21cccffb2b1c47c79eb20fd17d04cf82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:36:31.592445 8280 tablet_service.cc:1467] Processing CreateTablet for tablet 11c314b7fcb545068f6f73dce1bd1cab (DEFAULT_TABLE table=test-table [id=21cccffb2b1c47c79eb20fd17d04cf82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:36:31.593290 8140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11c314b7fcb545068f6f73dce1bd1cab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:36:31.593226 8419 tablet_service.cc:1467] Processing CreateTablet for tablet 11c314b7fcb545068f6f73dce1bd1cab (DEFAULT_TABLE table=test-table [id=21cccffb2b1c47c79eb20fd17d04cf82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:36:31.593832 8280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11c314b7fcb545068f6f73dce1bd1cab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:36:31.594587 8419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11c314b7fcb545068f6f73dce1bd1cab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:36:31.605189 8513 tablet_bootstrap.cc:492] T 11c314b7fcb545068f6f73dce1bd1cab P 3515e5dbdb114e0dbe8c96b2af448ce3: Bootstrap starting.
I20260620 23:36:31.608829 8514 tablet_bootstrap.cc:492] T 11c314b7fcb545068f6f73dce1bd1cab P e347312aaa774f81b9f0232b2f7f1d8d: Bootstrap starting.
I20260620 23:36:31.610037 8513 tablet_bootstrap.cc:654] T 11c314b7fcb545068f6f73dce1bd1cab P 3515e5dbdb114e0dbe8c96b2af448ce3: Neither blocks nor log segments found. Creating new log.
I20260620 23:36:31.611459 8513 log.cc:824] T 11c314b7fcb545068f6f73dce1bd1cab P 3515e5dbdb114e0dbe8c96b2af448ce3: Log is configured to *not* fsync() on all Append() calls
I20260620 23:36:31.611699 8515 tablet_bootstrap.cc:492] T 11c314b7fcb545068f6f73dce1bd1cab P 86d9648f36394a698d81a8dc587f77c1: Bootstrap starting.
I20260620 23:36:31.612579 8514 tablet_bootstrap.cc:654] T 11c314b7fcb545068f6f73dce1bd1cab P e347312aaa774f81b9f0232b2f7f1d8d: Neither blocks nor log segments found. Creating new log.
I20260620 23:36:31.613760 8514 log.cc:824] T 11c314b7fcb545068f6f73dce1bd1cab P e347312aaa774f81b9f0232b2f7f1d8d: Log is configured to *not* fsync() on all Append() calls
I20260620 23:36:31.615298 8515 tablet_bootstrap.cc:654] T 11c314b7fcb545068f6f73dce1bd1cab P 86d9648f36394a698d81a8dc587f77c1: Neither blocks nor log segments found. Creating new log.
I20260620 23:36:31.615355 8513 tablet_bootstrap.cc:492] T 11c314b7fcb545068f6f73dce1bd1cab P 3515e5dbdb114e0dbe8c96b2af448ce3: No bootstrap required, opened a new log
I20260620 23:36:31.616055 8513 ts_tablet_manager.cc:1397] T 11c314b7fcb545068f6f73dce1bd1cab P 3515e5dbdb114e0dbe8c96b2af448ce3: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260620 23:36:31.616578 8515 log.cc:824] T 11c314b7fcb545068f6f73dce1bd1cab P 86d9648f36394a698d81a8dc587f77c1: Log is configured to *not* fsync() on all Append() calls
I20260620 23:36:31.616803 8514 tablet_bootstrap.cc:492] T 11c314b7fcb545068f6f73dce1bd1cab P e347312aaa774f81b9f0232b2f7f1d8d: No bootstrap required, opened a new log
I20260620 23:36:31.617064 8514 ts_tablet_manager.cc:1397] T 11c314b7fcb545068f6f73dce1bd1cab P e347312aaa774f81b9f0232b2f7f1d8d: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260620 23:36:31.619671 8515 tablet_bootstrap.cc:492] T 11c314b7fcb545068f6f73dce1bd1cab P 86d9648f36394a698d81a8dc587f77c1: No bootstrap required, opened a new log
I20260620 23:36:31.619930 8515 ts_tablet_manager.cc:1397] T 11c314b7fcb545068f6f73dce1bd1cab P 86d9648f36394a698d81a8dc587f77c1: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 23:36:31.623591 8513 raft_consensus.cc:348] T 11c314b7fcb545068f6f73dce1bd1cab P 3515e5dbdb114e0dbe8c96b2af448ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40283 } } peers { permanent_uuid: "86d9648f36394a698d81a8dc587f77c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40371 } } peers { permanent_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44049 } }
I20260620 23:36:31.624050 8513 raft_consensus.cc:374] T 11c314b7fcb545068f6f73dce1bd1cab P 3515e5dbdb114e0dbe8c96b2af448ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:36:31.624191 8513 raft_consensus.cc:729] T 11c314b7fcb545068f6f73dce1bd1cab P 3515e5dbdb114e0dbe8c96b2af448ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3515e5dbdb114e0dbe8c96b2af448ce3, State: Initialized, Role: FOLLOWER
I20260620 23:36:31.624950 8513 consensus_queue.cc:260] T 11c314b7fcb545068f6f73dce1bd1cab P 3515e5dbdb114e0dbe8c96b2af448ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40283 } } peers { permanent_uuid: "86d9648f36394a698d81a8dc587f77c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40371 } } peers { permanent_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44049 } }
I20260620 23:36:31.627641 8514 raft_consensus.cc:348] T 11c314b7fcb545068f6f73dce1bd1cab P e347312aaa774f81b9f0232b2f7f1d8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40283 } } peers { permanent_uuid: "86d9648f36394a698d81a8dc587f77c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40371 } } peers { permanent_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44049 } }
I20260620 23:36:31.628171 8514 raft_consensus.cc:374] T 11c314b7fcb545068f6f73dce1bd1cab P e347312aaa774f81b9f0232b2f7f1d8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:36:31.628314 8514 raft_consensus.cc:729] T 11c314b7fcb545068f6f73dce1bd1cab P e347312aaa774f81b9f0232b2f7f1d8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e347312aaa774f81b9f0232b2f7f1d8d, State: Initialized, Role: FOLLOWER
I20260620 23:36:31.628680 8513 ts_tablet_manager.cc:1428] T 11c314b7fcb545068f6f73dce1bd1cab P 3515e5dbdb114e0dbe8c96b2af448ce3: Time spent starting tablet: real 0.012s user 0.010s sys 0.000s
I20260620 23:36:31.629125 8514 consensus_queue.cc:260] T 11c314b7fcb545068f6f73dce1bd1cab P e347312aaa774f81b9f0232b2f7f1d8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40283 } } peers { permanent_uuid: "86d9648f36394a698d81a8dc587f77c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40371 } } peers { permanent_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44049 } }
I20260620 23:36:31.630245 8350 heartbeater.cc:503] Master 127.0.97.62:34977 was elected leader, sending a full tablet report...
I20260620 23:36:31.630873 8514 ts_tablet_manager.cc:1428] T 11c314b7fcb545068f6f73dce1bd1cab P e347312aaa774f81b9f0232b2f7f1d8d: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260620 23:36:31.632982 8515 raft_consensus.cc:348] T 11c314b7fcb545068f6f73dce1bd1cab P 86d9648f36394a698d81a8dc587f77c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40283 } } peers { permanent_uuid: "86d9648f36394a698d81a8dc587f77c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40371 } } peers { permanent_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44049 } }
I20260620 23:36:31.633460 8515 raft_consensus.cc:374] T 11c314b7fcb545068f6f73dce1bd1cab P 86d9648f36394a698d81a8dc587f77c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:36:31.633575 8515 raft_consensus.cc:729] T 11c314b7fcb545068f6f73dce1bd1cab P 86d9648f36394a698d81a8dc587f77c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86d9648f36394a698d81a8dc587f77c1, State: Initialized, Role: FOLLOWER
I20260620 23:36:31.634222 8515 consensus_queue.cc:260] T 11c314b7fcb545068f6f73dce1bd1cab P 86d9648f36394a698d81a8dc587f77c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40283 } } peers { permanent_uuid: "86d9648f36394a698d81a8dc587f77c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40371 } } peers { permanent_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44049 } }
I20260620 23:36:31.635313 8489 heartbeater.cc:503] Master 127.0.97.62:34977 was elected leader, sending a full tablet report...
I20260620 23:36:31.635923 8515 ts_tablet_manager.cc:1428] T 11c314b7fcb545068f6f73dce1bd1cab P 86d9648f36394a698d81a8dc587f77c1: Time spent starting tablet: real 0.016s user 0.013s sys 0.000s
W20260620 23:36:31.673561 8351 tablet.cc:2369] T 11c314b7fcb545068f6f73dce1bd1cab P e347312aaa774f81b9f0232b2f7f1d8d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:36:31.715169 8490 tablet.cc:2369] T 11c314b7fcb545068f6f73dce1bd1cab P 86d9648f36394a698d81a8dc587f77c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:36:31.793737 8519 raft_consensus.cc:482] T 11c314b7fcb545068f6f73dce1bd1cab P 3515e5dbdb114e0dbe8c96b2af448ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:36:31.794082 8519 raft_consensus.cc:504] T 11c314b7fcb545068f6f73dce1bd1cab P 3515e5dbdb114e0dbe8c96b2af448ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40283 } } peers { permanent_uuid: "86d9648f36394a698d81a8dc587f77c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40371 } } peers { permanent_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44049 } }
I20260620 23:36:31.795812 8519 leader_election.cc:288] T 11c314b7fcb545068f6f73dce1bd1cab P 3515e5dbdb114e0dbe8c96b2af448ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86d9648f36394a698d81a8dc587f77c1 (127.0.97.3:40371), e347312aaa774f81b9f0232b2f7f1d8d (127.0.97.2:44049)
I20260620 23:36:31.802552 8510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:31.796297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37569 (local address 127.0.97.3:40371)
0620 23:36:31.796561 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:31.796568 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:31.796918 (+ 350us) server_negotiation.cc:407] Connection header received
0620 23:36:31.797249 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:31.797254 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:31.797324 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:31.797375 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:36:31.798225 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:31.798961 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:31.800769 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:31.801545 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:31.801706 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:31.801820 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:31.802137 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 23:36:31.802271 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260620 23:36:31.802722 8512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:31.797017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49959 (local address 127.0.97.2:44049)
0620 23:36:31.797213 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:31.797219 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:31.797574 (+ 355us) server_negotiation.cc:407] Connection header received
0620 23:36:31.797708 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:31.797713 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:31.797788 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:31.797888 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:36:31.798456 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:31.799252 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:31.800993 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:31.801740 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:31.801899 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:31.802023 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:31.802326 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 23:36:31.802450 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 23:36:31.802996 8522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:31.796180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40371 (local address 127.0.97.1:37569)
0620 23:36:31.796786 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:31.796802 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:31.797046 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:31.797563 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:31.797567 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:31.797596 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:36:31.798075 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:31.798086 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:31.799094 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:31.799100 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:31.800612 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:31.800624 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:31.801759 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:31.801765 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:31.801938 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:31.801966 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 23:36:31.802118 (+ 152us) client_negotiation.cc:240] Negotiation successful
0620 23:36:31.802271 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"mutex_wait_us":1,"thread_start_us":283,"threads_started":1}
I20260620 23:36:31.803169 8523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:31.796874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44049 (local address 127.0.97.1:49959)
0620 23:36:31.797468 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:31.797482 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:31.797588 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:31.797908 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:31.797911 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:31.797927 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:36:31.798258 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:31.798273 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:31.799387 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:31.799393 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:31.800814 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:31.800825 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:31.801979 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:31.801984 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:31.802158 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:31.802181 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 23:36:31.802292 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 23:36:31.802422 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"mutex_wait_us":22,"thread_start_us":300,"threads_started":1}
I20260620 23:36:31.804196 8305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11c314b7fcb545068f6f73dce1bd1cab" candidate_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" is_pre_election: true
I20260620 23:36:31.804188 8439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11c314b7fcb545068f6f73dce1bd1cab" candidate_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86d9648f36394a698d81a8dc587f77c1" is_pre_election: true
I20260620 23:36:31.804797 8439 raft_consensus.cc:2454] T 11c314b7fcb545068f6f73dce1bd1cab P 86d9648f36394a698d81a8dc587f77c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3515e5dbdb114e0dbe8c96b2af448ce3 in term 0.
I20260620 23:36:31.804793 8305 raft_consensus.cc:2454] T 11c314b7fcb545068f6f73dce1bd1cab P e347312aaa774f81b9f0232b2f7f1d8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3515e5dbdb114e0dbe8c96b2af448ce3 in term 0.
I20260620 23:36:31.805477 8093 leader_election.cc:302] T 11c314b7fcb545068f6f73dce1bd1cab P 3515e5dbdb114e0dbe8c96b2af448ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3515e5dbdb114e0dbe8c96b2af448ce3, 86d9648f36394a698d81a8dc587f77c1; no voters:
I20260620 23:36:31.805837 8519 raft_consensus.cc:2781] T 11c314b7fcb545068f6f73dce1bd1cab P 3515e5dbdb114e0dbe8c96b2af448ce3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:36:31.805982 8519 raft_consensus.cc:482] T 11c314b7fcb545068f6f73dce1bd1cab P 3515e5dbdb114e0dbe8c96b2af448ce3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:36:31.806093 8519 raft_consensus.cc:3037] T 11c314b7fcb545068f6f73dce1bd1cab P 3515e5dbdb114e0dbe8c96b2af448ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:36:31.808403 8519 raft_consensus.cc:504] T 11c314b7fcb545068f6f73dce1bd1cab P 3515e5dbdb114e0dbe8c96b2af448ce3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40283 } } peers { permanent_uuid: "86d9648f36394a698d81a8dc587f77c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40371 } } peers { permanent_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44049 } }
I20260620 23:36:31.809103 8519 leader_election.cc:288] T 11c314b7fcb545068f6f73dce1bd1cab P 3515e5dbdb114e0dbe8c96b2af448ce3 [CANDIDATE]: Term 1 election: Requested vote from peers 86d9648f36394a698d81a8dc587f77c1 (127.0.97.3:40371), e347312aaa774f81b9f0232b2f7f1d8d (127.0.97.2:44049)
I20260620 23:36:31.809572 8439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11c314b7fcb545068f6f73dce1bd1cab" candidate_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86d9648f36394a698d81a8dc587f77c1"
I20260620 23:36:31.809640 8305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11c314b7fcb545068f6f73dce1bd1cab" candidate_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e347312aaa774f81b9f0232b2f7f1d8d"
I20260620 23:36:31.809777 8439 raft_consensus.cc:3037] T 11c314b7fcb545068f6f73dce1bd1cab P 86d9648f36394a698d81a8dc587f77c1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:36:31.809891 8305 raft_consensus.cc:3037] T 11c314b7fcb545068f6f73dce1bd1cab P e347312aaa774f81b9f0232b2f7f1d8d [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:36:31.812145 8439 raft_consensus.cc:2454] T 11c314b7fcb545068f6f73dce1bd1cab P 86d9648f36394a698d81a8dc587f77c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3515e5dbdb114e0dbe8c96b2af448ce3 in term 1.
I20260620 23:36:31.812152 8305 raft_consensus.cc:2454] T 11c314b7fcb545068f6f73dce1bd1cab P e347312aaa774f81b9f0232b2f7f1d8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3515e5dbdb114e0dbe8c96b2af448ce3 in term 1.
I20260620 23:36:31.812572 8093 leader_election.cc:302] T 11c314b7fcb545068f6f73dce1bd1cab P 3515e5dbdb114e0dbe8c96b2af448ce3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3515e5dbdb114e0dbe8c96b2af448ce3, 86d9648f36394a698d81a8dc587f77c1; no voters:
I20260620 23:36:31.812886 8519 raft_consensus.cc:2781] T 11c314b7fcb545068f6f73dce1bd1cab P 3515e5dbdb114e0dbe8c96b2af448ce3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:36:31.813416 8519 raft_consensus.cc:686] T 11c314b7fcb545068f6f73dce1bd1cab P 3515e5dbdb114e0dbe8c96b2af448ce3 [term 1 LEADER]: Becoming Leader. State: Replica: 3515e5dbdb114e0dbe8c96b2af448ce3, State: Running, Role: LEADER
I20260620 23:36:31.814234 8519 consensus_queue.cc:237] T 11c314b7fcb545068f6f73dce1bd1cab P 3515e5dbdb114e0dbe8c96b2af448ce3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40283 } } peers { permanent_uuid: "86d9648f36394a698d81a8dc587f77c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40371 } } peers { permanent_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44049 } }
I20260620 23:36:31.818691 8003 catalog_manager.cc:5495] T 11c314b7fcb545068f6f73dce1bd1cab P 3515e5dbdb114e0dbe8c96b2af448ce3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3515e5dbdb114e0dbe8c96b2af448ce3 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "86d9648f36394a698d81a8dc587f77c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44049 } health_report { overall_health: UNKNOWN } } }
W20260620 23:36:31.849699 8211 tablet.cc:2369] T 11c314b7fcb545068f6f73dce1bd1cab P 3515e5dbdb114e0dbe8c96b2af448ce3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:36:31.883559 8508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:31.877903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41482 (local address 127.0.97.1:40283)
0620 23:36:31.878254 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:31.878259 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:31.878279 (+ 20us) server_negotiation.cc:407] Connection header received
0620 23:36:31.878374 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:31.878377 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:31.878466 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:31.878653 (+ 187us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:36:31.879002 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:31.879714 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:31.881110 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:31.881528 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:31.881695 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:31.882201 (+ 506us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:36:31.882847 (+ 646us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:36:31.883019 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:31.883143 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 23:36:31.883221 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235}
I20260620 23:36:31.891882 8439 raft_consensus.cc:1261] T 11c314b7fcb545068f6f73dce1bd1cab P 86d9648f36394a698d81a8dc587f77c1 [term 1 FOLLOWER]: Refusing update from remote peer 3515e5dbdb114e0dbe8c96b2af448ce3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:36:31.891888 8305 raft_consensus.cc:1261] T 11c314b7fcb545068f6f73dce1bd1cab P e347312aaa774f81b9f0232b2f7f1d8d [term 1 FOLLOWER]: Refusing update from remote peer 3515e5dbdb114e0dbe8c96b2af448ce3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:36:31.892872 8519 consensus_queue.cc:1035] T 11c314b7fcb545068f6f73dce1bd1cab P 3515e5dbdb114e0dbe8c96b2af448ce3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86d9648f36394a698d81a8dc587f77c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:36:31.893180 8524 consensus_queue.cc:1035] T 11c314b7fcb545068f6f73dce1bd1cab P 3515e5dbdb114e0dbe8c96b2af448ce3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:36:31.920759 8002 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:36:31.921069 8002 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:36:31.925197 8002 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 09f79b3ac58944c1be69f25d4e293a77: Sending DeleteTablet for 3 replicas of tablet 11c314b7fcb545068f6f73dce1bd1cab
I20260620 23:36:31.926510 8419 tablet_service.cc:1514] Processing DeleteTablet for tablet 11c314b7fcb545068f6f73dce1bd1cab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:36:31 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36820
I20260620 23:36:31.927413 8280 tablet_service.cc:1514] Processing DeleteTablet for tablet 11c314b7fcb545068f6f73dce1bd1cab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:36:31 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42826
I20260620 23:36:31.927489 8003 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40872:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 23:36:31.927805 8140 tablet_service.cc:1514] Processing DeleteTablet for tablet 11c314b7fcb545068f6f73dce1bd1cab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:36:31 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41472
W20260620 23:36:31.928246 8003 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:36:31.938252 8279 tablet_service.cc:1467] Processing CreateTablet for tablet a655cbe5565f44afaf2b84a73cfee08d (DEFAULT_TABLE table=test-table [id=c6b035e90a5a4a93a298a04137883009]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:36:31.938364 8418 tablet_service.cc:1467] Processing CreateTablet for tablet a655cbe5565f44afaf2b84a73cfee08d (DEFAULT_TABLE table=test-table [id=c6b035e90a5a4a93a298a04137883009]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:36:31.938894 8279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a655cbe5565f44afaf2b84a73cfee08d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:36:31.938905 8418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a655cbe5565f44afaf2b84a73cfee08d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:36:31.942310 8537 tablet_replica.cc:330] T 11c314b7fcb545068f6f73dce1bd1cab P 86d9648f36394a698d81a8dc587f77c1: stopping tablet replica
I20260620 23:36:31.942755 8537 raft_consensus.cc:2229] T 11c314b7fcb545068f6f73dce1bd1cab P 86d9648f36394a698d81a8dc587f77c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:36:31.946063 8514 tablet_bootstrap.cc:492] T a655cbe5565f44afaf2b84a73cfee08d P e347312aaa774f81b9f0232b2f7f1d8d: Bootstrap starting.
I20260620 23:36:31.947304 8139 tablet_service.cc:1467] Processing CreateTablet for tablet a655cbe5565f44afaf2b84a73cfee08d (DEFAULT_TABLE table=test-table [id=c6b035e90a5a4a93a298a04137883009]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:36:31.948503 8139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a655cbe5565f44afaf2b84a73cfee08d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:36:31.948629 8514 tablet_bootstrap.cc:654] T a655cbe5565f44afaf2b84a73cfee08d P e347312aaa774f81b9f0232b2f7f1d8d: Neither blocks nor log segments found. Creating new log.
I20260620 23:36:31.951252 8537 raft_consensus.cc:2258] T 11c314b7fcb545068f6f73dce1bd1cab P 86d9648f36394a698d81a8dc587f77c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:36:31.955235 8515 tablet_bootstrap.cc:492] T a655cbe5565f44afaf2b84a73cfee08d P 86d9648f36394a698d81a8dc587f77c1: Bootstrap starting.
I20260620 23:36:31.955323 8537 ts_tablet_manager.cc:1905] T 11c314b7fcb545068f6f73dce1bd1cab P 86d9648f36394a698d81a8dc587f77c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:36:31.957720 8515 tablet_bootstrap.cc:654] T a655cbe5565f44afaf2b84a73cfee08d P 86d9648f36394a698d81a8dc587f77c1: Neither blocks nor log segments found. Creating new log.
I20260620 23:36:31.961804 8539 tablet_replica.cc:330] T 11c314b7fcb545068f6f73dce1bd1cab P 3515e5dbdb114e0dbe8c96b2af448ce3: stopping tablet replica
I20260620 23:36:31.961897 8538 tablet_replica.cc:330] T 11c314b7fcb545068f6f73dce1bd1cab P e347312aaa774f81b9f0232b2f7f1d8d: stopping tablet replica
I20260620 23:36:31.962298 8539 raft_consensus.cc:2229] T 11c314b7fcb545068f6f73dce1bd1cab P 3515e5dbdb114e0dbe8c96b2af448ce3 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:36:31.962306 8538 raft_consensus.cc:2229] T 11c314b7fcb545068f6f73dce1bd1cab P e347312aaa774f81b9f0232b2f7f1d8d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:36:31.962688 8537 ts_tablet_manager.cc:1918] T 11c314b7fcb545068f6f73dce1bd1cab P 86d9648f36394a698d81a8dc587f77c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:36:31.962857 8537 log.cc:1196] T 11c314b7fcb545068f6f73dce1bd1cab P 86d9648f36394a698d81a8dc587f77c1: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/ts-2/wal/wals/11c314b7fcb545068f6f73dce1bd1cab
I20260620 23:36:31.963357 8537 ts_tablet_manager.cc:1939] T 11c314b7fcb545068f6f73dce1bd1cab P 86d9648f36394a698d81a8dc587f77c1: Deleting consensus metadata
I20260620 23:36:31.964471 8513 tablet_bootstrap.cc:492] T a655cbe5565f44afaf2b84a73cfee08d P 3515e5dbdb114e0dbe8c96b2af448ce3: Bootstrap starting.
I20260620 23:36:31.965180 8515 tablet_bootstrap.cc:492] T a655cbe5565f44afaf2b84a73cfee08d P 86d9648f36394a698d81a8dc587f77c1: No bootstrap required, opened a new log
I20260620 23:36:31.965342 8515 ts_tablet_manager.cc:1397] T a655cbe5565f44afaf2b84a73cfee08d P 86d9648f36394a698d81a8dc587f77c1: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.004s
I20260620 23:36:31.965366 7990 catalog_manager.cc:4841] TS 86d9648f36394a698d81a8dc587f77c1 (127.0.97.3:40371): tablet 11c314b7fcb545068f6f73dce1bd1cab (table test-table [id=21cccffb2b1c47c79eb20fd17d04cf82]) successfully deleted
I20260620 23:36:31.963836 8539 raft_consensus.cc:2258] T 11c314b7fcb545068f6f73dce1bd1cab P 3515e5dbdb114e0dbe8c96b2af448ce3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:36:31.966205 8513 tablet_bootstrap.cc:654] T a655cbe5565f44afaf2b84a73cfee08d P 3515e5dbdb114e0dbe8c96b2af448ce3: Neither blocks nor log segments found. Creating new log.
I20260620 23:36:31.966120 8515 raft_consensus.cc:348] T a655cbe5565f44afaf2b84a73cfee08d P 86d9648f36394a698d81a8dc587f77c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40283 } } peers { permanent_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44049 } } peers { permanent_uuid: "86d9648f36394a698d81a8dc587f77c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40371 } }
I20260620 23:36:31.967180 8514 tablet_bootstrap.cc:492] T a655cbe5565f44afaf2b84a73cfee08d P e347312aaa774f81b9f0232b2f7f1d8d: No bootstrap required, opened a new log
I20260620 23:36:31.967346 8514 ts_tablet_manager.cc:1397] T a655cbe5565f44afaf2b84a73cfee08d P e347312aaa774f81b9f0232b2f7f1d8d: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.013s
I20260620 23:36:31.967653 8515 raft_consensus.cc:374] T a655cbe5565f44afaf2b84a73cfee08d P 86d9648f36394a698d81a8dc587f77c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:36:31.967768 8515 raft_consensus.cc:729] T a655cbe5565f44afaf2b84a73cfee08d P 86d9648f36394a698d81a8dc587f77c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86d9648f36394a698d81a8dc587f77c1, State: Initialized, Role: FOLLOWER
I20260620 23:36:31.968225 8514 raft_consensus.cc:348] T a655cbe5565f44afaf2b84a73cfee08d P e347312aaa774f81b9f0232b2f7f1d8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40283 } } peers { permanent_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44049 } } peers { permanent_uuid: "86d9648f36394a698d81a8dc587f77c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40371 } }
I20260620 23:36:31.968236 8515 consensus_queue.cc:260] T a655cbe5565f44afaf2b84a73cfee08d P 86d9648f36394a698d81a8dc587f77c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40283 } } peers { permanent_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44049 } } peers { permanent_uuid: "86d9648f36394a698d81a8dc587f77c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40371 } }
I20260620 23:36:31.968444 8514 raft_consensus.cc:374] T a655cbe5565f44afaf2b84a73cfee08d P e347312aaa774f81b9f0232b2f7f1d8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:36:31.968550 8514 raft_consensus.cc:729] T a655cbe5565f44afaf2b84a73cfee08d P e347312aaa774f81b9f0232b2f7f1d8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e347312aaa774f81b9f0232b2f7f1d8d, State: Initialized, Role: FOLLOWER
I20260620 23:36:31.968873 8514 consensus_queue.cc:260] T a655cbe5565f44afaf2b84a73cfee08d P e347312aaa774f81b9f0232b2f7f1d8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40283 } } peers { permanent_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44049 } } peers { permanent_uuid: "86d9648f36394a698d81a8dc587f77c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40371 } }
I20260620 23:36:31.969465 8514 ts_tablet_manager.cc:1428] T a655cbe5565f44afaf2b84a73cfee08d P e347312aaa774f81b9f0232b2f7f1d8d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260620 23:36:31.969951 8538 raft_consensus.cc:2258] T 11c314b7fcb545068f6f73dce1bd1cab P e347312aaa774f81b9f0232b2f7f1d8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:36:31.970582 8515 ts_tablet_manager.cc:1428] T a655cbe5565f44afaf2b84a73cfee08d P 86d9648f36394a698d81a8dc587f77c1: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260620 23:36:31.974058 8513 tablet_bootstrap.cc:492] T a655cbe5565f44afaf2b84a73cfee08d P 3515e5dbdb114e0dbe8c96b2af448ce3: No bootstrap required, opened a new log
I20260620 23:36:31.974210 8513 ts_tablet_manager.cc:1397] T a655cbe5565f44afaf2b84a73cfee08d P 3515e5dbdb114e0dbe8c96b2af448ce3: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260620 23:36:31.975024 8513 raft_consensus.cc:348] T a655cbe5565f44afaf2b84a73cfee08d P 3515e5dbdb114e0dbe8c96b2af448ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40283 } } peers { permanent_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44049 } } peers { permanent_uuid: "86d9648f36394a698d81a8dc587f77c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40371 } }
I20260620 23:36:31.975255 8513 raft_consensus.cc:374] T a655cbe5565f44afaf2b84a73cfee08d P 3515e5dbdb114e0dbe8c96b2af448ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:36:31.975343 8513 raft_consensus.cc:729] T a655cbe5565f44afaf2b84a73cfee08d P 3515e5dbdb114e0dbe8c96b2af448ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3515e5dbdb114e0dbe8c96b2af448ce3, State: Initialized, Role: FOLLOWER
I20260620 23:36:31.975643 8513 consensus_queue.cc:260] T a655cbe5565f44afaf2b84a73cfee08d P 3515e5dbdb114e0dbe8c96b2af448ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40283 } } peers { permanent_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44049 } } peers { permanent_uuid: "86d9648f36394a698d81a8dc587f77c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40371 } }
I20260620 23:36:31.976562 8538 ts_tablet_manager.cc:1905] T 11c314b7fcb545068f6f73dce1bd1cab P e347312aaa774f81b9f0232b2f7f1d8d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:36:31.977005 8513 ts_tablet_manager.cc:1428] T a655cbe5565f44afaf2b84a73cfee08d P 3515e5dbdb114e0dbe8c96b2af448ce3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 23:36:31.978543 8539 ts_tablet_manager.cc:1905] T 11c314b7fcb545068f6f73dce1bd1cab P 3515e5dbdb114e0dbe8c96b2af448ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:36:31.981387 8519 raft_consensus.cc:482] T a655cbe5565f44afaf2b84a73cfee08d P 3515e5dbdb114e0dbe8c96b2af448ce3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:36:31.981562 8519 raft_consensus.cc:504] T a655cbe5565f44afaf2b84a73cfee08d P 3515e5dbdb114e0dbe8c96b2af448ce3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40283 } } peers { permanent_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44049 } } peers { permanent_uuid: "86d9648f36394a698d81a8dc587f77c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40371 } }
I20260620 23:36:31.982285 8519 leader_election.cc:288] T a655cbe5565f44afaf2b84a73cfee08d P 3515e5dbdb114e0dbe8c96b2af448ce3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e347312aaa774f81b9f0232b2f7f1d8d (127.0.97.2:44049), 86d9648f36394a698d81a8dc587f77c1 (127.0.97.3:40371)
I20260620 23:36:31.983212 8305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a655cbe5565f44afaf2b84a73cfee08d" candidate_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" is_pre_election: true
I20260620 23:36:31.983472 8305 raft_consensus.cc:2454] T a655cbe5565f44afaf2b84a73cfee08d P e347312aaa774f81b9f0232b2f7f1d8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3515e5dbdb114e0dbe8c96b2af448ce3 in term 0.
I20260620 23:36:31.984102 8439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a655cbe5565f44afaf2b84a73cfee08d" candidate_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86d9648f36394a698d81a8dc587f77c1" is_pre_election: true
I20260620 23:36:31.984345 8439 raft_consensus.cc:2454] T a655cbe5565f44afaf2b84a73cfee08d P 86d9648f36394a698d81a8dc587f77c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3515e5dbdb114e0dbe8c96b2af448ce3 in term 0.
I20260620 23:36:31.984997 8093 leader_election.cc:302] T a655cbe5565f44afaf2b84a73cfee08d P 3515e5dbdb114e0dbe8c96b2af448ce3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3515e5dbdb114e0dbe8c96b2af448ce3, 86d9648f36394a698d81a8dc587f77c1; no voters:
I20260620 23:36:31.985301 8538 ts_tablet_manager.cc:1918] T 11c314b7fcb545068f6f73dce1bd1cab P e347312aaa774f81b9f0232b2f7f1d8d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:36:31.985564 8538 log.cc:1196] T 11c314b7fcb545068f6f73dce1bd1cab P e347312aaa774f81b9f0232b2f7f1d8d: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/ts-1/wal/wals/11c314b7fcb545068f6f73dce1bd1cab
I20260620 23:36:31.985883 8519 raft_consensus.cc:2781] T a655cbe5565f44afaf2b84a73cfee08d P 3515e5dbdb114e0dbe8c96b2af448ce3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:36:31.986012 8519 raft_consensus.cc:482] T a655cbe5565f44afaf2b84a73cfee08d P 3515e5dbdb114e0dbe8c96b2af448ce3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:36:31.986011 8539 ts_tablet_manager.cc:1918] T 11c314b7fcb545068f6f73dce1bd1cab P 3515e5dbdb114e0dbe8c96b2af448ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:36:31.986135 8519 raft_consensus.cc:3037] T a655cbe5565f44afaf2b84a73cfee08d P 3515e5dbdb114e0dbe8c96b2af448ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:36:31.986212 8539 log.cc:1196] T 11c314b7fcb545068f6f73dce1bd1cab P 3515e5dbdb114e0dbe8c96b2af448ce3: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/ts-0/wal/wals/11c314b7fcb545068f6f73dce1bd1cab
I20260620 23:36:31.987437 8539 ts_tablet_manager.cc:1939] T 11c314b7fcb545068f6f73dce1bd1cab P 3515e5dbdb114e0dbe8c96b2af448ce3: Deleting consensus metadata
I20260620 23:36:31.987680 8538 ts_tablet_manager.cc:1939] T 11c314b7fcb545068f6f73dce1bd1cab P e347312aaa774f81b9f0232b2f7f1d8d: Deleting consensus metadata
I20260620 23:36:31.989320 7990 catalog_manager.cc:4841] TS 3515e5dbdb114e0dbe8c96b2af448ce3 (127.0.97.1:40283): tablet 11c314b7fcb545068f6f73dce1bd1cab (table test-table [id=21cccffb2b1c47c79eb20fd17d04cf82]) successfully deleted
I20260620 23:36:31.989643 7990 catalog_manager.cc:4841] TS e347312aaa774f81b9f0232b2f7f1d8d (127.0.97.2:44049): tablet 11c314b7fcb545068f6f73dce1bd1cab (table test-table [id=21cccffb2b1c47c79eb20fd17d04cf82]) successfully deleted
I20260620 23:36:31.989804 8519 raft_consensus.cc:504] T a655cbe5565f44afaf2b84a73cfee08d P 3515e5dbdb114e0dbe8c96b2af448ce3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40283 } } peers { permanent_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44049 } } peers { permanent_uuid: "86d9648f36394a698d81a8dc587f77c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40371 } }
I20260620 23:36:31.990496 8519 leader_election.cc:288] T a655cbe5565f44afaf2b84a73cfee08d P 3515e5dbdb114e0dbe8c96b2af448ce3 [CANDIDATE]: Term 1 election: Requested vote from peers e347312aaa774f81b9f0232b2f7f1d8d (127.0.97.2:44049), 86d9648f36394a698d81a8dc587f77c1 (127.0.97.3:40371)
I20260620 23:36:31.991084 8305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a655cbe5565f44afaf2b84a73cfee08d" candidate_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e347312aaa774f81b9f0232b2f7f1d8d"
I20260620 23:36:31.991289 8305 raft_consensus.cc:3037] T a655cbe5565f44afaf2b84a73cfee08d P e347312aaa774f81b9f0232b2f7f1d8d [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:36:31.991225 8439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a655cbe5565f44afaf2b84a73cfee08d" candidate_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86d9648f36394a698d81a8dc587f77c1"
I20260620 23:36:31.991459 8439 raft_consensus.cc:3037] T a655cbe5565f44afaf2b84a73cfee08d P 86d9648f36394a698d81a8dc587f77c1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:36:31.993326 8305 raft_consensus.cc:2454] T a655cbe5565f44afaf2b84a73cfee08d P e347312aaa774f81b9f0232b2f7f1d8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3515e5dbdb114e0dbe8c96b2af448ce3 in term 1.
I20260620 23:36:31.993800 8093 leader_election.cc:302] T a655cbe5565f44afaf2b84a73cfee08d P 3515e5dbdb114e0dbe8c96b2af448ce3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3515e5dbdb114e0dbe8c96b2af448ce3, e347312aaa774f81b9f0232b2f7f1d8d; no voters:
I20260620 23:36:31.994133 8519 raft_consensus.cc:2781] T a655cbe5565f44afaf2b84a73cfee08d P 3515e5dbdb114e0dbe8c96b2af448ce3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:36:31.994298 8519 raft_consensus.cc:686] T a655cbe5565f44afaf2b84a73cfee08d P 3515e5dbdb114e0dbe8c96b2af448ce3 [term 1 LEADER]: Becoming Leader. State: Replica: 3515e5dbdb114e0dbe8c96b2af448ce3, State: Running, Role: LEADER
I20260620 23:36:31.994578 8519 consensus_queue.cc:237] T a655cbe5565f44afaf2b84a73cfee08d P 3515e5dbdb114e0dbe8c96b2af448ce3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40283 } } peers { permanent_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44049 } } peers { permanent_uuid: "86d9648f36394a698d81a8dc587f77c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40371 } }
I20260620 23:36:31.995205 8439 raft_consensus.cc:2454] T a655cbe5565f44afaf2b84a73cfee08d P 86d9648f36394a698d81a8dc587f77c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3515e5dbdb114e0dbe8c96b2af448ce3 in term 1.
I20260620 23:36:31.997437 8001 catalog_manager.cc:5495] T a655cbe5565f44afaf2b84a73cfee08d P 3515e5dbdb114e0dbe8c96b2af448ce3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3515e5dbdb114e0dbe8c96b2af448ce3 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "86d9648f36394a698d81a8dc587f77c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40371 } health_report { overall_health: UNKNOWN } } }
Jun 20 23:36:32 dist-test-slave-tcgx krb5kdc[7958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998589, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 23:36:32.052462 8549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:32.029763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34977 (local address 127.0.0.1:40888)
0620 23:36:32.030670 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:32.030688 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:32.030835 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:32.031315 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:32.031319 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:32.032264 (+ 945us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:32.032713 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:32.032728 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:32.034329 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:32.034341 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:32.035747 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:32.035759 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:32.036895 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:32.036900 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:32.037012 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:32.043074 (+ 6062us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:32.043096 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:32.045450 (+ 2354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:36:32.049816 (+ 4366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:32.049828 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:32.049834 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:32.050261 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:32.050763 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:32.050771 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:32.050774 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:32.050997 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:32.051639 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:36:32.051648 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:36:32.051798 (+ 150us) client_negotiation.cc:769] Sending connection context
0620 23:36:32.051930 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 23:36:32.052145 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":736,"thread_start_us":551,"threads_started":1}
I20260620 23:36:32.052824 8238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:32.029763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40888 (local address 127.0.97.62:34977)
0620 23:36:32.030111 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:32.030121 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:32.030804 (+ 683us) server_negotiation.cc:407] Connection header received
0620 23:36:32.031003 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:32.031010 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:32.031101 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:32.031238 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:32.032893 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:32.034148 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:32.035906 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:32.036715 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:32.036855 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:32.045705 (+ 8850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:32.045714 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:32.045720 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:32.045766 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:32.049663 (+ 3897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:32.050389 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:32.050396 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:32.050400 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:32.050606 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:32.051112 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:32.051118 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:32.051123 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:32.051508 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:32.051679 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:32.052095 (+ 416us) server_negotiation.cc:299] Negotiation successful
0620 23:36:32.052338 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20260620 23:36:32.058678 8001 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40888:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 23:36:32.060109 8001 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:36:32.067451 8139 tablet_service.cc:1467] Processing CreateTablet for tablet 94838844204540e1bfb7a18833ddf14f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=111af16fa0a34ff2a93f0069e6b9ed52]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 23:36:32.068045 8139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94838844204540e1bfb7a18833ddf14f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:36:32.068157 8418 tablet_service.cc:1467] Processing CreateTablet for tablet 94838844204540e1bfb7a18833ddf14f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=111af16fa0a34ff2a93f0069e6b9ed52]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 23:36:32.068078 8280 tablet_service.cc:1467] Processing CreateTablet for tablet 94838844204540e1bfb7a18833ddf14f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=111af16fa0a34ff2a93f0069e6b9ed52]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 23:36:32.068742 8280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94838844204540e1bfb7a18833ddf14f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:36:32.068779 8418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94838844204540e1bfb7a18833ddf14f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:36:32.075583 8513 tablet_bootstrap.cc:492] T 94838844204540e1bfb7a18833ddf14f P 3515e5dbdb114e0dbe8c96b2af448ce3: Bootstrap starting.
I20260620 23:36:32.075747 8514 tablet_bootstrap.cc:492] T 94838844204540e1bfb7a18833ddf14f P e347312aaa774f81b9f0232b2f7f1d8d: Bootstrap starting.
I20260620 23:36:32.077420 8513 tablet_bootstrap.cc:654] T 94838844204540e1bfb7a18833ddf14f P 3515e5dbdb114e0dbe8c96b2af448ce3: Neither blocks nor log segments found. Creating new log.
I20260620 23:36:32.077410 8514 tablet_bootstrap.cc:654] T 94838844204540e1bfb7a18833ddf14f P e347312aaa774f81b9f0232b2f7f1d8d: Neither blocks nor log segments found. Creating new log.
I20260620 23:36:32.077802 8515 tablet_bootstrap.cc:492] T 94838844204540e1bfb7a18833ddf14f P 86d9648f36394a698d81a8dc587f77c1: Bootstrap starting.
I20260620 23:36:32.079613 8515 tablet_bootstrap.cc:654] T 94838844204540e1bfb7a18833ddf14f P 86d9648f36394a698d81a8dc587f77c1: Neither blocks nor log segments found. Creating new log.
I20260620 23:36:32.089496 8514 tablet_bootstrap.cc:492] T 94838844204540e1bfb7a18833ddf14f P e347312aaa774f81b9f0232b2f7f1d8d: No bootstrap required, opened a new log
I20260620 23:36:32.089843 8514 ts_tablet_manager.cc:1397] T 94838844204540e1bfb7a18833ddf14f P e347312aaa774f81b9f0232b2f7f1d8d: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.006s
I20260620 23:36:32.091001 8514 raft_consensus.cc:348] T 94838844204540e1bfb7a18833ddf14f P e347312aaa774f81b9f0232b2f7f1d8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40283 } } peers { permanent_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44049 } } peers { permanent_uuid: "86d9648f36394a698d81a8dc587f77c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40371 } }
I20260620 23:36:32.091465 8514 raft_consensus.cc:374] T 94838844204540e1bfb7a18833ddf14f P e347312aaa774f81b9f0232b2f7f1d8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:36:32.091612 8514 raft_consensus.cc:729] T 94838844204540e1bfb7a18833ddf14f P e347312aaa774f81b9f0232b2f7f1d8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e347312aaa774f81b9f0232b2f7f1d8d, State: Initialized, Role: FOLLOWER
I20260620 23:36:32.092121 8514 consensus_queue.cc:260] T 94838844204540e1bfb7a18833ddf14f P e347312aaa774f81b9f0232b2f7f1d8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40283 } } peers { permanent_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44049 } } peers { permanent_uuid: "86d9648f36394a698d81a8dc587f77c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40371 } }
I20260620 23:36:32.093083 8515 tablet_bootstrap.cc:492] T 94838844204540e1bfb7a18833ddf14f P 86d9648f36394a698d81a8dc587f77c1: No bootstrap required, opened a new log
I20260620 23:36:32.092844 8520 tablet_replica.cc:439] T 94838844204540e1bfb7a18833ddf14f P e347312aaa774f81b9f0232b2f7f1d8d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40283 } } peers { permanent_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44049 } } peers { permanent_uuid: "86d9648f36394a698d81a8dc587f77c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40371 } } }
I20260620 23:36:32.093319 8520 tablet_replica.cc:442] T 94838844204540e1bfb7a18833ddf14f P e347312aaa774f81b9f0232b2f7f1d8d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:36:32.094374 8515 ts_tablet_manager.cc:1397] T 94838844204540e1bfb7a18833ddf14f P 86d9648f36394a698d81a8dc587f77c1: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.006s
I20260620 23:36:32.095249 8515 raft_consensus.cc:348] T 94838844204540e1bfb7a18833ddf14f P 86d9648f36394a698d81a8dc587f77c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40283 } } peers { permanent_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44049 } } peers { permanent_uuid: "86d9648f36394a698d81a8dc587f77c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40371 } }
I20260620 23:36:32.095511 8515 raft_consensus.cc:374] T 94838844204540e1bfb7a18833ddf14f P 86d9648f36394a698d81a8dc587f77c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:36:32.095584 8515 raft_consensus.cc:729] T 94838844204540e1bfb7a18833ddf14f P 86d9648f36394a698d81a8dc587f77c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86d9648f36394a698d81a8dc587f77c1, State: Initialized, Role: FOLLOWER
I20260620 23:36:32.095882 8515 consensus_queue.cc:260] T 94838844204540e1bfb7a18833ddf14f P 86d9648f36394a698d81a8dc587f77c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40283 } } peers { permanent_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44049 } } peers { permanent_uuid: "86d9648f36394a698d81a8dc587f77c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40371 } }
I20260620 23:36:32.096347 8521 tablet_replica.cc:439] T 94838844204540e1bfb7a18833ddf14f P 86d9648f36394a698d81a8dc587f77c1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40283 } } peers { permanent_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44049 } } peers { permanent_uuid: "86d9648f36394a698d81a8dc587f77c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40371 } } }
I20260620 23:36:32.096673 8521 tablet_replica.cc:442] T 94838844204540e1bfb7a18833ddf14f P 86d9648f36394a698d81a8dc587f77c1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:36:32.097267 8514 ts_tablet_manager.cc:1428] T 94838844204540e1bfb7a18833ddf14f P e347312aaa774f81b9f0232b2f7f1d8d: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260620 23:36:32.098006 8515 ts_tablet_manager.cc:1428] T 94838844204540e1bfb7a18833ddf14f P 86d9648f36394a698d81a8dc587f77c1: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260620 23:36:32.103299 8513 tablet_bootstrap.cc:492] T 94838844204540e1bfb7a18833ddf14f P 3515e5dbdb114e0dbe8c96b2af448ce3: No bootstrap required, opened a new log
I20260620 23:36:32.103525 8513 ts_tablet_manager.cc:1397] T 94838844204540e1bfb7a18833ddf14f P 3515e5dbdb114e0dbe8c96b2af448ce3: Time spent bootstrapping tablet: real 0.028s user 0.014s sys 0.012s
I20260620 23:36:32.104192 8513 raft_consensus.cc:348] T 94838844204540e1bfb7a18833ddf14f P 3515e5dbdb114e0dbe8c96b2af448ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40283 } } peers { permanent_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44049 } } peers { permanent_uuid: "86d9648f36394a698d81a8dc587f77c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40371 } }
I20260620 23:36:32.104427 8513 raft_consensus.cc:374] T 94838844204540e1bfb7a18833ddf14f P 3515e5dbdb114e0dbe8c96b2af448ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:36:32.104513 8513 raft_consensus.cc:729] T 94838844204540e1bfb7a18833ddf14f P 3515e5dbdb114e0dbe8c96b2af448ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3515e5dbdb114e0dbe8c96b2af448ce3, State: Initialized, Role: FOLLOWER
I20260620 23:36:32.104826 8513 consensus_queue.cc:260] T 94838844204540e1bfb7a18833ddf14f P 3515e5dbdb114e0dbe8c96b2af448ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40283 } } peers { permanent_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44049 } } peers { permanent_uuid: "86d9648f36394a698d81a8dc587f77c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40371 } }
I20260620 23:36:32.105466 8513 ts_tablet_manager.cc:1428] T 94838844204540e1bfb7a18833ddf14f P 3515e5dbdb114e0dbe8c96b2af448ce3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:36:32.105310 8519 tablet_replica.cc:439] T 94838844204540e1bfb7a18833ddf14f P 3515e5dbdb114e0dbe8c96b2af448ce3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40283 } } peers { permanent_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44049 } } peers { permanent_uuid: "86d9648f36394a698d81a8dc587f77c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40371 } } }
I20260620 23:36:32.105703 8519 tablet_replica.cc:442] T 94838844204540e1bfb7a18833ddf14f P 3515e5dbdb114e0dbe8c96b2af448ce3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:36:32.205827 8520 raft_consensus.cc:482] T 94838844204540e1bfb7a18833ddf14f P e347312aaa774f81b9f0232b2f7f1d8d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:36:32.206048 8520 raft_consensus.cc:504] T 94838844204540e1bfb7a18833ddf14f P e347312aaa774f81b9f0232b2f7f1d8d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40283 } } peers { permanent_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44049 } } peers { permanent_uuid: "86d9648f36394a698d81a8dc587f77c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40371 } }
I20260620 23:36:32.208125 8520 leader_election.cc:288] T 94838844204540e1bfb7a18833ddf14f P e347312aaa774f81b9f0232b2f7f1d8d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3515e5dbdb114e0dbe8c96b2af448ce3 (127.0.97.1:40283), 86d9648f36394a698d81a8dc587f77c1 (127.0.97.3:40371)
I20260620 23:36:32.217088 8508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:32.208507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40781 (local address 127.0.97.1:40283)
0620 23:36:32.208768 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:32.208776 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:32.209534 (+ 758us) server_negotiation.cc:407] Connection header received
0620 23:36:32.209656 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:32.209665 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:32.209782 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:32.209914 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:36:32.210687 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:32.211999 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:32.214335 (+ 2336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:32.215858 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:32.216033 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:32.216166 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:32.216553 (+ 387us) server_negotiation.cc:299] Negotiation successful
0620 23:36:32.216719 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 23:36:32.217532 8554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:32.209374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40371 (local address 127.0.97.2:40081)
0620 23:36:32.210690 (+ 1316us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:32.210710 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:32.210846 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:32.211234 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:32.211239 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:32.211269 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:36:32.211660 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:32.211670 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:32.213236 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:32.213243 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:32.215092 (+ 1849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:32.215103 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:32.216667 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:32.216673 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:32.216855 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:32.216893 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 23:36:32.216992 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 23:36:32.217151 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1150,"thread_start_us":938,"threads_started":1}
I20260620 23:36:32.218026 8553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:32.208501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40283 (local address 127.0.97.2:40781)
0620 23:36:32.209210 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:32.209234 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:32.209378 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:32.209936 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:32.209941 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:32.209980 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:36:32.210439 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:32.210463 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:32.212143 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:32.212151 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:32.214130 (+ 1979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:32.214146 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:32.216114 (+ 1968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:32.216131 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:32.216341 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:32.216383 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 23:36:32.217399 (+ 1016us) client_negotiation.cc:240] Negotiation successful
0620 23:36:32.217556 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"mutex_wait_us":1,"thread_start_us":378,"threads_started":1}
I20260620 23:36:32.218189 8510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:32.209583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40081 (local address 127.0.97.3:40371)
0620 23:36:32.209788 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:32.209796 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:32.210811 (+ 1015us) server_negotiation.cc:407] Connection header received
0620 23:36:32.210988 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:32.210996 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:32.211094 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:32.211198 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:36:32.211846 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:32.213082 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:32.215255 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:32.216260 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:32.216389 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:32.216516 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:32.217048 (+ 532us) server_negotiation.cc:299] Negotiation successful
0620 23:36:32.217439 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 23:36:32.218761 8439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94838844204540e1bfb7a18833ddf14f" candidate_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86d9648f36394a698d81a8dc587f77c1" is_pre_election: true
I20260620 23:36:32.219091 8439 raft_consensus.cc:2454] T 94838844204540e1bfb7a18833ddf14f P 86d9648f36394a698d81a8dc587f77c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e347312aaa774f81b9f0232b2f7f1d8d in term 0.
I20260620 23:36:32.218978 8160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94838844204540e1bfb7a18833ddf14f" candidate_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" is_pre_election: true
I20260620 23:36:32.219583 8160 raft_consensus.cc:2454] T 94838844204540e1bfb7a18833ddf14f P 3515e5dbdb114e0dbe8c96b2af448ce3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e347312aaa774f81b9f0232b2f7f1d8d in term 0.
I20260620 23:36:32.219692 8233 leader_election.cc:302] T 94838844204540e1bfb7a18833ddf14f P e347312aaa774f81b9f0232b2f7f1d8d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86d9648f36394a698d81a8dc587f77c1, e347312aaa774f81b9f0232b2f7f1d8d; no voters:
I20260620 23:36:32.220204 8520 raft_consensus.cc:2781] T 94838844204540e1bfb7a18833ddf14f P e347312aaa774f81b9f0232b2f7f1d8d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:36:32.220434 8520 raft_consensus.cc:482] T 94838844204540e1bfb7a18833ddf14f P e347312aaa774f81b9f0232b2f7f1d8d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:36:32.220546 8520 raft_consensus.cc:3037] T 94838844204540e1bfb7a18833ddf14f P e347312aaa774f81b9f0232b2f7f1d8d [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:36:32.222774 8520 raft_consensus.cc:504] T 94838844204540e1bfb7a18833ddf14f P e347312aaa774f81b9f0232b2f7f1d8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40283 } } peers { permanent_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44049 } } peers { permanent_uuid: "86d9648f36394a698d81a8dc587f77c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40371 } }
I20260620 23:36:32.223521 8520 leader_election.cc:288] T 94838844204540e1bfb7a18833ddf14f P e347312aaa774f81b9f0232b2f7f1d8d [CANDIDATE]: Term 1 election: Requested vote from peers 3515e5dbdb114e0dbe8c96b2af448ce3 (127.0.97.1:40283), 86d9648f36394a698d81a8dc587f77c1 (127.0.97.3:40371)
I20260620 23:36:32.224020 8160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94838844204540e1bfb7a18833ddf14f" candidate_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3"
I20260620 23:36:32.224157 8439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94838844204540e1bfb7a18833ddf14f" candidate_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86d9648f36394a698d81a8dc587f77c1"
I20260620 23:36:32.224267 8160 raft_consensus.cc:3037] T 94838844204540e1bfb7a18833ddf14f P 3515e5dbdb114e0dbe8c96b2af448ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:36:32.224416 8439 raft_consensus.cc:3037] T 94838844204540e1bfb7a18833ddf14f P 86d9648f36394a698d81a8dc587f77c1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:36:32.227058 8160 raft_consensus.cc:2454] T 94838844204540e1bfb7a18833ddf14f P 3515e5dbdb114e0dbe8c96b2af448ce3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e347312aaa774f81b9f0232b2f7f1d8d in term 1.
I20260620 23:36:32.227579 8233 leader_election.cc:302] T 94838844204540e1bfb7a18833ddf14f P e347312aaa774f81b9f0232b2f7f1d8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3515e5dbdb114e0dbe8c96b2af448ce3, e347312aaa774f81b9f0232b2f7f1d8d; no voters:
I20260620 23:36:32.227902 8520 raft_consensus.cc:2781] T 94838844204540e1bfb7a18833ddf14f P e347312aaa774f81b9f0232b2f7f1d8d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:36:32.227885 8439 raft_consensus.cc:2454] T 94838844204540e1bfb7a18833ddf14f P 86d9648f36394a698d81a8dc587f77c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e347312aaa774f81b9f0232b2f7f1d8d in term 1.
I20260620 23:36:32.228549 8520 raft_consensus.cc:686] T 94838844204540e1bfb7a18833ddf14f P e347312aaa774f81b9f0232b2f7f1d8d [term 1 LEADER]: Becoming Leader. State: Replica: e347312aaa774f81b9f0232b2f7f1d8d, State: Running, Role: LEADER
I20260620 23:36:32.228984 8520 consensus_queue.cc:237] T 94838844204540e1bfb7a18833ddf14f P e347312aaa774f81b9f0232b2f7f1d8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40283 } } peers { permanent_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44049 } } peers { permanent_uuid: "86d9648f36394a698d81a8dc587f77c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40371 } }
I20260620 23:36:32.232154 8555 tablet_replica.cc:439] T 94838844204540e1bfb7a18833ddf14f P e347312aaa774f81b9f0232b2f7f1d8d: TxnStatusTablet state changed. Reason: New leader e347312aaa774f81b9f0232b2f7f1d8d. Latest consensus state: current_term: 1 leader_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40283 } } peers { permanent_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44049 } } peers { permanent_uuid: "86d9648f36394a698d81a8dc587f77c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40371 } } }
I20260620 23:36:32.232555 8555 tablet_replica.cc:442] T 94838844204540e1bfb7a18833ddf14f P e347312aaa774f81b9f0232b2f7f1d8d: This TxnStatusTablet replica's current role is: LEADER
I20260620 23:36:32.233268 8557 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 23:36:32.233075 8003 catalog_manager.cc:5495] T 94838844204540e1bfb7a18833ddf14f P e347312aaa774f81b9f0232b2f7f1d8d reported cstate change: term changed from 0 to 1, leader changed from <none> to e347312aaa774f81b9f0232b2f7f1d8d (127.0.97.2). New cstate: current_term: 1 leader_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "86d9648f36394a698d81a8dc587f77c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40371 } health_report { overall_health: UNKNOWN } } }
I20260620 23:36:32.233546 8557 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 23:36:32.282805 8549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:32.276636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44049 (local address 127.0.0.1:42830)
0620 23:36:32.276970 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:32.276985 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:32.277155 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:32.277638 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:32.277642 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:32.277654 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:36:32.277845 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:32.277854 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:32.278910 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:32.278918 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:32.279971 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:32.279982 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:32.280829 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:32.280835 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:32.281074 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:32.281131 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:36:32.282235 (+ 1104us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:36:32.282257 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 23:36:32.282362 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 23:36:32.282502 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182}
I20260620 23:36:32.283660 8512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:32.276704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42830 (local address 127.0.97.2:44049)
0620 23:36:32.276890 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:32.276898 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:32.277060 (+ 162us) server_negotiation.cc:407] Connection header received
0620 23:36:32.277363 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:32.277368 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:32.277477 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:32.277620 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:36:32.277977 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:32.278782 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:32.280111 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:32.280669 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:32.280811 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:32.281363 (+ 552us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:36:32.282122 (+ 759us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:36:32.282287 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:32.283279 (+ 992us) server_negotiation.cc:299] Negotiation successful
0620 23:36:32.283358 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 23:36:32.287846 8160 raft_consensus.cc:1261] T 94838844204540e1bfb7a18833ddf14f P 3515e5dbdb114e0dbe8c96b2af448ce3 [term 1 FOLLOWER]: Refusing update from remote peer e347312aaa774f81b9f0232b2f7f1d8d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:36:32.288647 8439 raft_consensus.cc:1261] T 94838844204540e1bfb7a18833ddf14f P 86d9648f36394a698d81a8dc587f77c1 [term 1 FOLLOWER]: Refusing update from remote peer e347312aaa774f81b9f0232b2f7f1d8d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:36:32.288692 8520 consensus_queue.cc:1035] T 94838844204540e1bfb7a18833ddf14f P e347312aaa774f81b9f0232b2f7f1d8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:36:32.289295 8555 consensus_queue.cc:1035] T 94838844204540e1bfb7a18833ddf14f P e347312aaa774f81b9f0232b2f7f1d8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "86d9648f36394a698d81a8dc587f77c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:36:32.302986 8519 tablet_replica.cc:439] T 94838844204540e1bfb7a18833ddf14f P 3515e5dbdb114e0dbe8c96b2af448ce3: TxnStatusTablet state changed. Reason: New leader e347312aaa774f81b9f0232b2f7f1d8d. Latest consensus state: current_term: 1 leader_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40283 } } peers { permanent_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44049 } } peers { permanent_uuid: "86d9648f36394a698d81a8dc587f77c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40371 } } }
I20260620 23:36:32.303388 8519 tablet_replica.cc:442] T 94838844204540e1bfb7a18833ddf14f P 3515e5dbdb114e0dbe8c96b2af448ce3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:36:32.304049 8521 tablet_replica.cc:439] T 94838844204540e1bfb7a18833ddf14f P 86d9648f36394a698d81a8dc587f77c1: TxnStatusTablet state changed. Reason: New leader e347312aaa774f81b9f0232b2f7f1d8d. Latest consensus state: current_term: 1 leader_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40283 } } peers { permanent_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44049 } } peers { permanent_uuid: "86d9648f36394a698d81a8dc587f77c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40371 } } }
I20260620 23:36:32.304322 8521 tablet_replica.cc:442] T 94838844204540e1bfb7a18833ddf14f P 86d9648f36394a698d81a8dc587f77c1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:36:32.306770 8520 tablet_replica.cc:439] T 94838844204540e1bfb7a18833ddf14f P e347312aaa774f81b9f0232b2f7f1d8d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40283 } } peers { permanent_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44049 } } peers { permanent_uuid: "86d9648f36394a698d81a8dc587f77c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40371 } } }
I20260620 23:36:32.307091 8520 tablet_replica.cc:442] T 94838844204540e1bfb7a18833ddf14f P e347312aaa774f81b9f0232b2f7f1d8d: This TxnStatusTablet replica's current role is: LEADER
I20260620 23:36:32.308491 8519 tablet_replica.cc:439] T 94838844204540e1bfb7a18833ddf14f P 3515e5dbdb114e0dbe8c96b2af448ce3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40283 } } peers { permanent_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44049 } } peers { permanent_uuid: "86d9648f36394a698d81a8dc587f77c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40371 } } }
I20260620 23:36:32.308786 8519 tablet_replica.cc:442] T 94838844204540e1bfb7a18833ddf14f P 3515e5dbdb114e0dbe8c96b2af448ce3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:36:32.311857 8529 mvcc.cc:205] Tried to move back new op lower bound from 7299066234002522112 to 7299066233773649920. Current Snapshot: MvccSnapshot[applied={T|T < 7299066234002522112 or (T in {7299066234002522112})}]
I20260620 23:36:32.342549 8520 tablet_replica.cc:439] T 94838844204540e1bfb7a18833ddf14f P e347312aaa774f81b9f0232b2f7f1d8d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40283 } } peers { permanent_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44049 } } peers { permanent_uuid: "86d9648f36394a698d81a8dc587f77c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40371 } } }
I20260620 23:36:32.342878 8520 tablet_replica.cc:442] T 94838844204540e1bfb7a18833ddf14f P e347312aaa774f81b9f0232b2f7f1d8d: This TxnStatusTablet replica's current role is: LEADER
I20260620 23:36:32.345955 8528 mvcc.cc:205] Tried to move back new op lower bound from 7299066234002522112 to 7299066233773649920. Current Snapshot: MvccSnapshot[applied={T|T < 7299066234002522112}]
I20260620 23:36:32.346588 8521 tablet_replica.cc:439] T 94838844204540e1bfb7a18833ddf14f P 86d9648f36394a698d81a8dc587f77c1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40283 } } peers { permanent_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44049 } } peers { permanent_uuid: "86d9648f36394a698d81a8dc587f77c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40371 } } }
I20260620 23:36:32.346870 8521 tablet_replica.cc:442] T 94838844204540e1bfb7a18833ddf14f P 86d9648f36394a698d81a8dc587f77c1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:36:32.375262 8512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:32.357425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55059 (local address 127.0.97.2:44049)
0620 23:36:32.357606 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:32.357613 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:32.367735 (+ 10122us) server_negotiation.cc:407] Connection header received
0620 23:36:32.367982 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:32.367990 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:32.368097 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:32.368216 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:36:32.369078 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:32.370295 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:32.372784 (+ 2489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:32.373468 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:32.373582 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:32.374364 (+ 782us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:36:32.374505 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:36:32.374573 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:32.374838 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 23:36:32.374928 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 23:36:32.376372 8566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:32.356965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44049 (local address 127.0.97.1:55059)
0620 23:36:32.367546 (+ 10581us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:32.367579 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:32.367788 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:32.368414 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:32.368419 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:32.368433 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:36:32.368861 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:32.368878 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:32.370698 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:32.370703 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:32.372565 (+ 1862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:32.372586 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:32.373813 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:32.373820 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:32.374047 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:32.374147 (+ 100us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:36:32.374656 (+ 509us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:36:32.374683 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 23:36:32.375795 (+ 1112us) client_negotiation.cc:240] Negotiation successful
0620 23:36:32.375950 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10311,"thread_start_us":10125,"threads_started":1}
I20260620 23:36:32.394197 8439 raft_consensus.cc:1261] T a655cbe5565f44afaf2b84a73cfee08d P 86d9648f36394a698d81a8dc587f77c1 [term 1 FOLLOWER]: Refusing update from remote peer 3515e5dbdb114e0dbe8c96b2af448ce3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:36:32.394315 8305 raft_consensus.cc:1261] T a655cbe5565f44afaf2b84a73cfee08d P e347312aaa774f81b9f0232b2f7f1d8d [term 1 FOLLOWER]: Refusing update from remote peer 3515e5dbdb114e0dbe8c96b2af448ce3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:36:32.395049 8532 consensus_queue.cc:1035] T a655cbe5565f44afaf2b84a73cfee08d P 3515e5dbdb114e0dbe8c96b2af448ce3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86d9648f36394a698d81a8dc587f77c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:36:32.395735 8519 consensus_queue.cc:1035] T a655cbe5565f44afaf2b84a73cfee08d P 3515e5dbdb114e0dbe8c96b2af448ce3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:36:32.411221 8527 mvcc.cc:205] Tried to move back new op lower bound from 7299066234438955008 to 7299066232812748800. Current Snapshot: MvccSnapshot[applied={T|T < 7299066234438955008}]
I20260620 23:36:32.449113 8508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:32.440262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42371 (local address 127.0.97.1:40283)
0620 23:36:32.440719 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:32.440728 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:32.442722 (+ 1994us) server_negotiation.cc:407] Connection header received
0620 23:36:32.442958 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:32.442965 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:32.443054 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:32.443158 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:36:32.443690 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:32.444821 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:32.446663 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:32.447225 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:32.447386 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:32.447795 (+ 409us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:36:32.447953 (+ 158us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:36:32.448074 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:32.448698 (+ 624us) server_negotiation.cc:299] Negotiation successful
0620 23:36:32.448787 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266}
I20260620 23:36:32.450259 8571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:32.439772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40283 (local address 127.0.97.2:42371)
0620 23:36:32.442597 (+ 2825us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:32.442614 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:32.442805 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:32.443171 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:32.443175 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:32.443187 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:36:32.443544 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:32.443553 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:32.444978 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:32.444983 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:32.446499 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:32.446512 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:32.447421 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:32.447430 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:32.447558 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:32.447615 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:36:32.448524 (+ 909us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:36:32.448542 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 23:36:32.449696 (+ 1154us) client_negotiation.cc:240] Negotiation successful
0620 23:36:32.449845 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2622,"thread_start_us":2457,"threads_started":1}
I20260620 23:36:32.518496 8003 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:36:32.518832 8003 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:36:32.522351 8003 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 09f79b3ac58944c1be69f25d4e293a77: Sending DeleteTablet for 3 replicas of tablet a655cbe5565f44afaf2b84a73cfee08d
I20260620 23:36:32.523345 8418 tablet_service.cc:1514] Processing DeleteTablet for tablet a655cbe5565f44afaf2b84a73cfee08d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:36:32 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36820
I20260620 23:36:32.524063 8280 tablet_service.cc:1514] Processing DeleteTablet for tablet a655cbe5565f44afaf2b84a73cfee08d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:36:32 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42826
I20260620 23:36:32.525750 8139 tablet_service.cc:1514] Processing DeleteTablet for tablet a655cbe5565f44afaf2b84a73cfee08d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:36:32 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41472
I20260620 23:36:32.531893 8575 tablet_replica.cc:330] T a655cbe5565f44afaf2b84a73cfee08d P e347312aaa774f81b9f0232b2f7f1d8d: stopping tablet replica
I20260620 23:36:32.532275 8575 raft_consensus.cc:2229] T a655cbe5565f44afaf2b84a73cfee08d P e347312aaa774f81b9f0232b2f7f1d8d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:36:32.532681 8575 raft_consensus.cc:2258] T a655cbe5565f44afaf2b84a73cfee08d P e347312aaa774f81b9f0232b2f7f1d8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:36:32.534013 8575 ts_tablet_manager.cc:1905] T a655cbe5565f44afaf2b84a73cfee08d P e347312aaa774f81b9f0232b2f7f1d8d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:36:32.548605 8575 ts_tablet_manager.cc:1918] T a655cbe5565f44afaf2b84a73cfee08d P e347312aaa774f81b9f0232b2f7f1d8d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:36:32.548907 8575 log.cc:1196] T a655cbe5565f44afaf2b84a73cfee08d P e347312aaa774f81b9f0232b2f7f1d8d: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/ts-1/wal/wals/a655cbe5565f44afaf2b84a73cfee08d
I20260620 23:36:32.549546 8575 ts_tablet_manager.cc:1939] T a655cbe5565f44afaf2b84a73cfee08d P e347312aaa774f81b9f0232b2f7f1d8d: Deleting consensus metadata
I20260620 23:36:32.550015 8576 tablet_replica.cc:330] T a655cbe5565f44afaf2b84a73cfee08d P 3515e5dbdb114e0dbe8c96b2af448ce3: stopping tablet replica
I20260620 23:36:32.550246 8576 raft_consensus.cc:2229] T a655cbe5565f44afaf2b84a73cfee08d P 3515e5dbdb114e0dbe8c96b2af448ce3 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:36:32.550609 8576 raft_consensus.cc:2258] T a655cbe5565f44afaf2b84a73cfee08d P 3515e5dbdb114e0dbe8c96b2af448ce3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:36:32.551076 7990 catalog_manager.cc:4841] TS e347312aaa774f81b9f0232b2f7f1d8d (127.0.97.2:44049): tablet a655cbe5565f44afaf2b84a73cfee08d (table test-table [id=c6b035e90a5a4a93a298a04137883009]) successfully deleted
I20260620 23:36:32.551692 8574 tablet_replica.cc:330] T a655cbe5565f44afaf2b84a73cfee08d P 86d9648f36394a698d81a8dc587f77c1: stopping tablet replica
I20260620 23:36:32.551882 8574 raft_consensus.cc:2229] T a655cbe5565f44afaf2b84a73cfee08d P 86d9648f36394a698d81a8dc587f77c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:36:32.552071 8574 raft_consensus.cc:2258] T a655cbe5565f44afaf2b84a73cfee08d P 86d9648f36394a698d81a8dc587f77c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:36:32.554138 8574 ts_tablet_manager.cc:1905] T a655cbe5565f44afaf2b84a73cfee08d P 86d9648f36394a698d81a8dc587f77c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:36:32.555249 8576 ts_tablet_manager.cc:1905] T a655cbe5565f44afaf2b84a73cfee08d P 3515e5dbdb114e0dbe8c96b2af448ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:36:32.562647 8576 ts_tablet_manager.cc:1918] T a655cbe5565f44afaf2b84a73cfee08d P 3515e5dbdb114e0dbe8c96b2af448ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:36:32.562647 8574 ts_tablet_manager.cc:1918] T a655cbe5565f44afaf2b84a73cfee08d P 86d9648f36394a698d81a8dc587f77c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:36:32.562811 8576 log.cc:1196] T a655cbe5565f44afaf2b84a73cfee08d P 3515e5dbdb114e0dbe8c96b2af448ce3: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/ts-0/wal/wals/a655cbe5565f44afaf2b84a73cfee08d
I20260620 23:36:32.562984 8574 log.cc:1196] T a655cbe5565f44afaf2b84a73cfee08d P 86d9648f36394a698d81a8dc587f77c1: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/ts-2/wal/wals/a655cbe5565f44afaf2b84a73cfee08d
I20260620 23:36:32.563452 8576 ts_tablet_manager.cc:1939] T a655cbe5565f44afaf2b84a73cfee08d P 3515e5dbdb114e0dbe8c96b2af448ce3: Deleting consensus metadata
I20260620 23:36:32.563455 8574 ts_tablet_manager.cc:1939] T a655cbe5565f44afaf2b84a73cfee08d P 86d9648f36394a698d81a8dc587f77c1: Deleting consensus metadata
I20260620 23:36:32.564627 7990 catalog_manager.cc:4841] TS 86d9648f36394a698d81a8dc587f77c1 (127.0.97.3:40371): tablet a655cbe5565f44afaf2b84a73cfee08d (table test-table [id=c6b035e90a5a4a93a298a04137883009]) successfully deleted
I20260620 23:36:32.564963 7990 catalog_manager.cc:4841] TS 3515e5dbdb114e0dbe8c96b2af448ce3 (127.0.97.1:40283): tablet a655cbe5565f44afaf2b84a73cfee08d (table test-table [id=c6b035e90a5a4a93a298a04137883009]) successfully deleted
Jun 20 23:36:32 dist-test-slave-tcgx krb5kdc[7958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 23:36:32 dist-test-slave-tcgx krb5kdc[7958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:36:32 dist-test-slave-tcgx krb5kdc[7958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 23:36:32.600773 8584 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:36:32.589339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40904 (local address 127.0.97.62:34977)
0620 23:36:32.589883 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:32.589889 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:32.590219 (+ 330us) server_negotiation.cc:407] Connection header received
0620 23:36:32.590373 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:32.590379 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:32.590461 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:32.590686 (+ 225us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:32.592037 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:32.593214 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:32.594435 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:32.595111 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:32.595258 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:32.600444 (+ 5186us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40904: BlockingRecv error: recv got EOF from 127.0.0.1:40904 (error 108)
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":254,"threads_started":1}
Jun 20 23:36:32 dist-test-slave-tcgx krb5kdc[7958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 23:36:32.619283 8584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:32.603643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40912 (local address 127.0.97.62:34977)
0620 23:36:32.603819 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:32.603825 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:32.604192 (+ 367us) server_negotiation.cc:407] Connection header received
0620 23:36:32.604342 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:32.604347 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:32.604423 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:32.604517 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:32.605796 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:32.606901 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:32.608343 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:32.608979 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:32.609115 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:32.613119 (+ 4004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:32.613126 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:32.613132 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:32.613178 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:32.616516 (+ 3338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:32.617203 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:32.617208 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:32.617212 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:32.617392 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:32.617771 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:32.617776 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:32.617779 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:32.618142 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:32.618410 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:32.618653 (+ 243us) server_negotiation.cc:299] Negotiation successful
0620 23:36:32.618885 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 23:36:32.623106 8003 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40912:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:36:32.623757 8003 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:36:32.630793 8139 tablet_service.cc:1467] Processing CreateTablet for tablet 54abdab7f29b4d05ad35dd5d65af6b33 (DEFAULT_TABLE table=test-table [id=8ea58ac87587464b952058e47743d9c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:36:32.631357 8139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54abdab7f29b4d05ad35dd5d65af6b33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:36:32.631639 8280 tablet_service.cc:1467] Processing CreateTablet for tablet 54abdab7f29b4d05ad35dd5d65af6b33 (DEFAULT_TABLE table=test-table [id=8ea58ac87587464b952058e47743d9c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:36:32.632079 8280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54abdab7f29b4d05ad35dd5d65af6b33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:36:32.637282 8590 tablet_bootstrap.cc:492] T 54abdab7f29b4d05ad35dd5d65af6b33 P e347312aaa774f81b9f0232b2f7f1d8d: Bootstrap starting.
I20260620 23:36:32.637220 8418 tablet_service.cc:1467] Processing CreateTablet for tablet 54abdab7f29b4d05ad35dd5d65af6b33 (DEFAULT_TABLE table=test-table [id=8ea58ac87587464b952058e47743d9c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:36:32.637699 8418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54abdab7f29b4d05ad35dd5d65af6b33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:36:32.639416 8590 tablet_bootstrap.cc:654] T 54abdab7f29b4d05ad35dd5d65af6b33 P e347312aaa774f81b9f0232b2f7f1d8d: Neither blocks nor log segments found. Creating new log.
I20260620 23:36:32.642969 8591 tablet_bootstrap.cc:492] T 54abdab7f29b4d05ad35dd5d65af6b33 P 3515e5dbdb114e0dbe8c96b2af448ce3: Bootstrap starting.
I20260620 23:36:32.644745 8591 tablet_bootstrap.cc:654] T 54abdab7f29b4d05ad35dd5d65af6b33 P 3515e5dbdb114e0dbe8c96b2af448ce3: Neither blocks nor log segments found. Creating new log.
I20260620 23:36:32.647897 8591 tablet_bootstrap.cc:492] T 54abdab7f29b4d05ad35dd5d65af6b33 P 3515e5dbdb114e0dbe8c96b2af448ce3: No bootstrap required, opened a new log
I20260620 23:36:32.648056 8591 ts_tablet_manager.cc:1397] T 54abdab7f29b4d05ad35dd5d65af6b33 P 3515e5dbdb114e0dbe8c96b2af448ce3: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260620 23:36:32.648938 8591 raft_consensus.cc:348] T 54abdab7f29b4d05ad35dd5d65af6b33 P 3515e5dbdb114e0dbe8c96b2af448ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40283 } } peers { permanent_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44049 } } peers { permanent_uuid: "86d9648f36394a698d81a8dc587f77c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40371 } }
I20260620 23:36:32.649191 8591 raft_consensus.cc:374] T 54abdab7f29b4d05ad35dd5d65af6b33 P 3515e5dbdb114e0dbe8c96b2af448ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:36:32.649345 8591 raft_consensus.cc:729] T 54abdab7f29b4d05ad35dd5d65af6b33 P 3515e5dbdb114e0dbe8c96b2af448ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3515e5dbdb114e0dbe8c96b2af448ce3, State: Initialized, Role: FOLLOWER
I20260620 23:36:32.649624 8591 consensus_queue.cc:260] T 54abdab7f29b4d05ad35dd5d65af6b33 P 3515e5dbdb114e0dbe8c96b2af448ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40283 } } peers { permanent_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44049 } } peers { permanent_uuid: "86d9648f36394a698d81a8dc587f77c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40371 } }
I20260620 23:36:32.651286 8594 tablet_bootstrap.cc:492] T 54abdab7f29b4d05ad35dd5d65af6b33 P 86d9648f36394a698d81a8dc587f77c1: Bootstrap starting.
I20260620 23:36:32.652670 8591 ts_tablet_manager.cc:1428] T 54abdab7f29b4d05ad35dd5d65af6b33 P 3515e5dbdb114e0dbe8c96b2af448ce3: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20260620 23:36:32.655777 8594 tablet_bootstrap.cc:654] T 54abdab7f29b4d05ad35dd5d65af6b33 P 86d9648f36394a698d81a8dc587f77c1: Neither blocks nor log segments found. Creating new log.
I20260620 23:36:32.659011 8590 tablet_bootstrap.cc:492] T 54abdab7f29b4d05ad35dd5d65af6b33 P e347312aaa774f81b9f0232b2f7f1d8d: No bootstrap required, opened a new log
I20260620 23:36:32.659013 8594 tablet_bootstrap.cc:492] T 54abdab7f29b4d05ad35dd5d65af6b33 P 86d9648f36394a698d81a8dc587f77c1: No bootstrap required, opened a new log
I20260620 23:36:32.659178 8594 ts_tablet_manager.cc:1397] T 54abdab7f29b4d05ad35dd5d65af6b33 P 86d9648f36394a698d81a8dc587f77c1: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.002s
I20260620 23:36:32.659183 8590 ts_tablet_manager.cc:1397] T 54abdab7f29b4d05ad35dd5d65af6b33 P e347312aaa774f81b9f0232b2f7f1d8d: Time spent bootstrapping tablet: real 0.022s user 0.011s sys 0.008s
I20260620 23:36:32.659992 8594 raft_consensus.cc:348] T 54abdab7f29b4d05ad35dd5d65af6b33 P 86d9648f36394a698d81a8dc587f77c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40283 } } peers { permanent_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44049 } } peers { permanent_uuid: "86d9648f36394a698d81a8dc587f77c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40371 } }
I20260620 23:36:32.659998 8590 raft_consensus.cc:348] T 54abdab7f29b4d05ad35dd5d65af6b33 P e347312aaa774f81b9f0232b2f7f1d8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40283 } } peers { permanent_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44049 } } peers { permanent_uuid: "86d9648f36394a698d81a8dc587f77c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40371 } }
I20260620 23:36:32.660259 8590 raft_consensus.cc:374] T 54abdab7f29b4d05ad35dd5d65af6b33 P e347312aaa774f81b9f0232b2f7f1d8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:36:32.660252 8594 raft_consensus.cc:374] T 54abdab7f29b4d05ad35dd5d65af6b33 P 86d9648f36394a698d81a8dc587f77c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:36:32.660364 8590 raft_consensus.cc:729] T 54abdab7f29b4d05ad35dd5d65af6b33 P e347312aaa774f81b9f0232b2f7f1d8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e347312aaa774f81b9f0232b2f7f1d8d, State: Initialized, Role: FOLLOWER
I20260620 23:36:32.660364 8594 raft_consensus.cc:729] T 54abdab7f29b4d05ad35dd5d65af6b33 P 86d9648f36394a698d81a8dc587f77c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86d9648f36394a698d81a8dc587f77c1, State: Initialized, Role: FOLLOWER
I20260620 23:36:32.660611 8594 consensus_queue.cc:260] T 54abdab7f29b4d05ad35dd5d65af6b33 P 86d9648f36394a698d81a8dc587f77c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40283 } } peers { permanent_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44049 } } peers { permanent_uuid: "86d9648f36394a698d81a8dc587f77c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40371 } }
I20260620 23:36:32.660619 8590 consensus_queue.cc:260] T 54abdab7f29b4d05ad35dd5d65af6b33 P e347312aaa774f81b9f0232b2f7f1d8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40283 } } peers { permanent_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44049 } } peers { permanent_uuid: "86d9648f36394a698d81a8dc587f77c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40371 } }
I20260620 23:36:32.661340 8594 ts_tablet_manager.cc:1428] T 54abdab7f29b4d05ad35dd5d65af6b33 P 86d9648f36394a698d81a8dc587f77c1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:36:32.661324 8590 ts_tablet_manager.cc:1428] T 54abdab7f29b4d05ad35dd5d65af6b33 P e347312aaa774f81b9f0232b2f7f1d8d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 23:36:32.782194 8562 raft_consensus.cc:482] T 54abdab7f29b4d05ad35dd5d65af6b33 P e347312aaa774f81b9f0232b2f7f1d8d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:36:32.782505 8562 raft_consensus.cc:504] T 54abdab7f29b4d05ad35dd5d65af6b33 P e347312aaa774f81b9f0232b2f7f1d8d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40283 } } peers { permanent_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44049 } } peers { permanent_uuid: "86d9648f36394a698d81a8dc587f77c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40371 } }
I20260620 23:36:32.783262 8562 leader_election.cc:288] T 54abdab7f29b4d05ad35dd5d65af6b33 P e347312aaa774f81b9f0232b2f7f1d8d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3515e5dbdb114e0dbe8c96b2af448ce3 (127.0.97.1:40283), 86d9648f36394a698d81a8dc587f77c1 (127.0.97.3:40371)
I20260620 23:36:32.783955 8438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54abdab7f29b4d05ad35dd5d65af6b33" candidate_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86d9648f36394a698d81a8dc587f77c1" is_pre_election: true
I20260620 23:36:32.783993 8160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54abdab7f29b4d05ad35dd5d65af6b33" candidate_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" is_pre_election: true
I20260620 23:36:32.784199 8438 raft_consensus.cc:2454] T 54abdab7f29b4d05ad35dd5d65af6b33 P 86d9648f36394a698d81a8dc587f77c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e347312aaa774f81b9f0232b2f7f1d8d in term 0.
I20260620 23:36:32.784237 8160 raft_consensus.cc:2454] T 54abdab7f29b4d05ad35dd5d65af6b33 P 3515e5dbdb114e0dbe8c96b2af448ce3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e347312aaa774f81b9f0232b2f7f1d8d in term 0.
I20260620 23:36:32.784641 8233 leader_election.cc:302] T 54abdab7f29b4d05ad35dd5d65af6b33 P e347312aaa774f81b9f0232b2f7f1d8d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86d9648f36394a698d81a8dc587f77c1, e347312aaa774f81b9f0232b2f7f1d8d; no voters:
I20260620 23:36:32.784914 8562 raft_consensus.cc:2781] T 54abdab7f29b4d05ad35dd5d65af6b33 P e347312aaa774f81b9f0232b2f7f1d8d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:36:32.785037 8562 raft_consensus.cc:482] T 54abdab7f29b4d05ad35dd5d65af6b33 P e347312aaa774f81b9f0232b2f7f1d8d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:36:32.785130 8562 raft_consensus.cc:3037] T 54abdab7f29b4d05ad35dd5d65af6b33 P e347312aaa774f81b9f0232b2f7f1d8d [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:36:32.788482 8562 raft_consensus.cc:504] T 54abdab7f29b4d05ad35dd5d65af6b33 P e347312aaa774f81b9f0232b2f7f1d8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40283 } } peers { permanent_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44049 } } peers { permanent_uuid: "86d9648f36394a698d81a8dc587f77c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40371 } }
I20260620 23:36:32.789215 8562 leader_election.cc:288] T 54abdab7f29b4d05ad35dd5d65af6b33 P e347312aaa774f81b9f0232b2f7f1d8d [CANDIDATE]: Term 1 election: Requested vote from peers 3515e5dbdb114e0dbe8c96b2af448ce3 (127.0.97.1:40283), 86d9648f36394a698d81a8dc587f77c1 (127.0.97.3:40371)
I20260620 23:36:32.789789 8160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54abdab7f29b4d05ad35dd5d65af6b33" candidate_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3"
I20260620 23:36:32.789831 8438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54abdab7f29b4d05ad35dd5d65af6b33" candidate_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86d9648f36394a698d81a8dc587f77c1"
I20260620 23:36:32.790050 8160 raft_consensus.cc:3037] T 54abdab7f29b4d05ad35dd5d65af6b33 P 3515e5dbdb114e0dbe8c96b2af448ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:36:32.790050 8438 raft_consensus.cc:3037] T 54abdab7f29b4d05ad35dd5d65af6b33 P 86d9648f36394a698d81a8dc587f77c1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:36:32.793032 8160 raft_consensus.cc:2454] T 54abdab7f29b4d05ad35dd5d65af6b33 P 3515e5dbdb114e0dbe8c96b2af448ce3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e347312aaa774f81b9f0232b2f7f1d8d in term 1.
I20260620 23:36:32.793032 8438 raft_consensus.cc:2454] T 54abdab7f29b4d05ad35dd5d65af6b33 P 86d9648f36394a698d81a8dc587f77c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e347312aaa774f81b9f0232b2f7f1d8d in term 1.
I20260620 23:36:32.793473 8233 leader_election.cc:302] T 54abdab7f29b4d05ad35dd5d65af6b33 P e347312aaa774f81b9f0232b2f7f1d8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3515e5dbdb114e0dbe8c96b2af448ce3, e347312aaa774f81b9f0232b2f7f1d8d; no voters:
I20260620 23:36:32.793809 8562 raft_consensus.cc:2781] T 54abdab7f29b4d05ad35dd5d65af6b33 P e347312aaa774f81b9f0232b2f7f1d8d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:36:32.793957 8562 raft_consensus.cc:686] T 54abdab7f29b4d05ad35dd5d65af6b33 P e347312aaa774f81b9f0232b2f7f1d8d [term 1 LEADER]: Becoming Leader. State: Replica: e347312aaa774f81b9f0232b2f7f1d8d, State: Running, Role: LEADER
I20260620 23:36:32.794232 8562 consensus_queue.cc:237] T 54abdab7f29b4d05ad35dd5d65af6b33 P e347312aaa774f81b9f0232b2f7f1d8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40283 } } peers { permanent_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44049 } } peers { permanent_uuid: "86d9648f36394a698d81a8dc587f77c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40371 } }
I20260620 23:36:32.797052 8002 catalog_manager.cc:5495] T 54abdab7f29b4d05ad35dd5d65af6b33 P e347312aaa774f81b9f0232b2f7f1d8d reported cstate change: term changed from 0 to 1, leader changed from <none> to e347312aaa774f81b9f0232b2f7f1d8d (127.0.97.2). New cstate: current_term: 1 leader_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "86d9648f36394a698d81a8dc587f77c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40371 } health_report { overall_health: UNKNOWN } } }
I20260620 23:36:32.811098 8512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:32.805795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42834 (local address 127.0.97.2:44049)
0620 23:36:32.805977 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:32.805985 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:32.806029 (+ 44us) server_negotiation.cc:407] Connection header received
0620 23:36:32.806266 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:32.806275 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:32.806351 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:32.806453 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:36:32.806810 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:32.807846 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:32.809307 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:32.809744 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:32.809884 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:32.810315 (+ 431us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:36:32.810432 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:36:32.810534 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:32.810675 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 23:36:32.810777 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 23:36:32.814100 8160 raft_consensus.cc:1261] T 54abdab7f29b4d05ad35dd5d65af6b33 P 3515e5dbdb114e0dbe8c96b2af448ce3 [term 1 FOLLOWER]: Refusing update from remote peer e347312aaa774f81b9f0232b2f7f1d8d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:36:32.814150 8438 raft_consensus.cc:1261] T 54abdab7f29b4d05ad35dd5d65af6b33 P 86d9648f36394a698d81a8dc587f77c1 [term 1 FOLLOWER]: Refusing update from remote peer e347312aaa774f81b9f0232b2f7f1d8d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:36:32.814663 8562 consensus_queue.cc:1035] T 54abdab7f29b4d05ad35dd5d65af6b33 P e347312aaa774f81b9f0232b2f7f1d8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:36:32.815017 8565 consensus_queue.cc:1035] T 54abdab7f29b4d05ad35dd5d65af6b33 P e347312aaa774f81b9f0232b2f7f1d8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "86d9648f36394a698d81a8dc587f77c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:36:32.829958 8002 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:36:32.830253 8002 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:36:32.833503 8002 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 09f79b3ac58944c1be69f25d4e293a77: Sending DeleteTablet for 3 replicas of tablet 54abdab7f29b4d05ad35dd5d65af6b33
I20260620 23:36:32.834439 8280 tablet_service.cc:1514] Processing DeleteTablet for tablet 54abdab7f29b4d05ad35dd5d65af6b33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:36:32 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42826
I20260620 23:36:32.834789 8575 tablet_replica.cc:330] T 54abdab7f29b4d05ad35dd5d65af6b33 P e347312aaa774f81b9f0232b2f7f1d8d: stopping tablet replica
I20260620 23:36:32.834772 8139 tablet_service.cc:1514] Processing DeleteTablet for tablet 54abdab7f29b4d05ad35dd5d65af6b33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:36:32 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41472
I20260620 23:36:32.835042 8575 raft_consensus.cc:2229] T 54abdab7f29b4d05ad35dd5d65af6b33 P e347312aaa774f81b9f0232b2f7f1d8d [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:36:32.835088 8576 tablet_replica.cc:330] T 54abdab7f29b4d05ad35dd5d65af6b33 P 3515e5dbdb114e0dbe8c96b2af448ce3: stopping tablet replica
I20260620 23:36:32.835266 8576 raft_consensus.cc:2229] T 54abdab7f29b4d05ad35dd5d65af6b33 P 3515e5dbdb114e0dbe8c96b2af448ce3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:36:32.835500 8576 raft_consensus.cc:2258] T 54abdab7f29b4d05ad35dd5d65af6b33 P 3515e5dbdb114e0dbe8c96b2af448ce3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:36:32.835601 8575 raft_consensus.cc:2258] T 54abdab7f29b4d05ad35dd5d65af6b33 P e347312aaa774f81b9f0232b2f7f1d8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:36:32.834892 8418 tablet_service.cc:1514] Processing DeleteTablet for tablet 54abdab7f29b4d05ad35dd5d65af6b33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:36:32 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36820
I20260620 23:36:32.836195 8574 tablet_replica.cc:330] T 54abdab7f29b4d05ad35dd5d65af6b33 P 86d9648f36394a698d81a8dc587f77c1: stopping tablet replica
I20260620 23:36:32.836418 8574 raft_consensus.cc:2229] T 54abdab7f29b4d05ad35dd5d65af6b33 P 86d9648f36394a698d81a8dc587f77c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:36:32.836644 8574 raft_consensus.cc:2258] T 54abdab7f29b4d05ad35dd5d65af6b33 P 86d9648f36394a698d81a8dc587f77c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:36:32.837011 8576 ts_tablet_manager.cc:1905] T 54abdab7f29b4d05ad35dd5d65af6b33 P 3515e5dbdb114e0dbe8c96b2af448ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:36:32.837677 8575 ts_tablet_manager.cc:1905] T 54abdab7f29b4d05ad35dd5d65af6b33 P e347312aaa774f81b9f0232b2f7f1d8d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:36:32.840632 8574 ts_tablet_manager.cc:1905] T 54abdab7f29b4d05ad35dd5d65af6b33 P 86d9648f36394a698d81a8dc587f77c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:36:32.843616 8576 ts_tablet_manager.cc:1918] T 54abdab7f29b4d05ad35dd5d65af6b33 P 3515e5dbdb114e0dbe8c96b2af448ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:36:32.843422 8002 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40912:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 23:36:32.843775 8576 log.cc:1196] T 54abdab7f29b4d05ad35dd5d65af6b33 P 3515e5dbdb114e0dbe8c96b2af448ce3: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/ts-0/wal/wals/54abdab7f29b4d05ad35dd5d65af6b33
I20260620 23:36:32.844177 8576 ts_tablet_manager.cc:1939] T 54abdab7f29b4d05ad35dd5d65af6b33 P 3515e5dbdb114e0dbe8c96b2af448ce3: Deleting consensus metadata
I20260620 23:36:32.846554 8574 ts_tablet_manager.cc:1918] T 54abdab7f29b4d05ad35dd5d65af6b33 P 86d9648f36394a698d81a8dc587f77c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:36:32.846725 8574 log.cc:1196] T 54abdab7f29b4d05ad35dd5d65af6b33 P 86d9648f36394a698d81a8dc587f77c1: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/ts-2/wal/wals/54abdab7f29b4d05ad35dd5d65af6b33
I20260620 23:36:32.845376 7990 catalog_manager.cc:4841] TS 3515e5dbdb114e0dbe8c96b2af448ce3 (127.0.97.1:40283): tablet 54abdab7f29b4d05ad35dd5d65af6b33 (table test-table [id=8ea58ac87587464b952058e47743d9c9]) successfully deleted
I20260620 23:36:32.847215 8574 ts_tablet_manager.cc:1939] T 54abdab7f29b4d05ad35dd5d65af6b33 P 86d9648f36394a698d81a8dc587f77c1: Deleting consensus metadata
W20260620 23:36:32.847851 8002 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:36:32.848181 7990 catalog_manager.cc:4841] TS 86d9648f36394a698d81a8dc587f77c1 (127.0.97.3:40371): tablet 54abdab7f29b4d05ad35dd5d65af6b33 (table test-table [id=8ea58ac87587464b952058e47743d9c9]) successfully deleted
I20260620 23:36:32.850618 8575 ts_tablet_manager.cc:1918] T 54abdab7f29b4d05ad35dd5d65af6b33 P e347312aaa774f81b9f0232b2f7f1d8d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:36:32.850807 8575 log.cc:1196] T 54abdab7f29b4d05ad35dd5d65af6b33 P e347312aaa774f81b9f0232b2f7f1d8d: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/ts-1/wal/wals/54abdab7f29b4d05ad35dd5d65af6b33
I20260620 23:36:32.851320 8575 ts_tablet_manager.cc:1939] T 54abdab7f29b4d05ad35dd5d65af6b33 P e347312aaa774f81b9f0232b2f7f1d8d: Deleting consensus metadata
I20260620 23:36:32.852954 7990 catalog_manager.cc:4841] TS e347312aaa774f81b9f0232b2f7f1d8d (127.0.97.2:44049): tablet 54abdab7f29b4d05ad35dd5d65af6b33 (table test-table [id=8ea58ac87587464b952058e47743d9c9]) successfully deleted
I20260620 23:36:32.855536 8139 tablet_service.cc:1467] Processing CreateTablet for tablet 02aaa73cb73f4632a129ce70a9d4555f (DEFAULT_TABLE table=test-table [id=0cc7083b34b844ffad9528e02ca8ce15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:36:32.855832 8280 tablet_service.cc:1467] Processing CreateTablet for tablet 02aaa73cb73f4632a129ce70a9d4555f (DEFAULT_TABLE table=test-table [id=0cc7083b34b844ffad9528e02ca8ce15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:36:32.855858 8418 tablet_service.cc:1467] Processing CreateTablet for tablet 02aaa73cb73f4632a129ce70a9d4555f (DEFAULT_TABLE table=test-table [id=0cc7083b34b844ffad9528e02ca8ce15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:36:32.855969 8139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02aaa73cb73f4632a129ce70a9d4555f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:36:32.856225 8418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02aaa73cb73f4632a129ce70a9d4555f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:36:32.856375 8280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02aaa73cb73f4632a129ce70a9d4555f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:36:32.862515 8591 tablet_bootstrap.cc:492] T 02aaa73cb73f4632a129ce70a9d4555f P 3515e5dbdb114e0dbe8c96b2af448ce3: Bootstrap starting.
I20260620 23:36:32.862967 8594 tablet_bootstrap.cc:492] T 02aaa73cb73f4632a129ce70a9d4555f P 86d9648f36394a698d81a8dc587f77c1: Bootstrap starting.
I20260620 23:36:32.864254 8591 tablet_bootstrap.cc:654] T 02aaa73cb73f4632a129ce70a9d4555f P 3515e5dbdb114e0dbe8c96b2af448ce3: Neither blocks nor log segments found. Creating new log.
I20260620 23:36:32.864427 8594 tablet_bootstrap.cc:654] T 02aaa73cb73f4632a129ce70a9d4555f P 86d9648f36394a698d81a8dc587f77c1: Neither blocks nor log segments found. Creating new log.
I20260620 23:36:32.866354 8590 tablet_bootstrap.cc:492] T 02aaa73cb73f4632a129ce70a9d4555f P e347312aaa774f81b9f0232b2f7f1d8d: Bootstrap starting.
I20260620 23:36:32.868052 8594 tablet_bootstrap.cc:492] T 02aaa73cb73f4632a129ce70a9d4555f P 86d9648f36394a698d81a8dc587f77c1: No bootstrap required, opened a new log
I20260620 23:36:32.868214 8594 ts_tablet_manager.cc:1397] T 02aaa73cb73f4632a129ce70a9d4555f P 86d9648f36394a698d81a8dc587f77c1: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 23:36:32.868240 8590 tablet_bootstrap.cc:654] T 02aaa73cb73f4632a129ce70a9d4555f P e347312aaa774f81b9f0232b2f7f1d8d: Neither blocks nor log segments found. Creating new log.
I20260620 23:36:32.868999 8594 raft_consensus.cc:348] T 02aaa73cb73f4632a129ce70a9d4555f P 86d9648f36394a698d81a8dc587f77c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40283 } } peers { permanent_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44049 } } peers { permanent_uuid: "86d9648f36394a698d81a8dc587f77c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40371 } }
I20260620 23:36:32.869264 8594 raft_consensus.cc:374] T 02aaa73cb73f4632a129ce70a9d4555f P 86d9648f36394a698d81a8dc587f77c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:36:32.869349 8594 raft_consensus.cc:729] T 02aaa73cb73f4632a129ce70a9d4555f P 86d9648f36394a698d81a8dc587f77c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86d9648f36394a698d81a8dc587f77c1, State: Initialized, Role: FOLLOWER
I20260620 23:36:32.869529 8591 tablet_bootstrap.cc:492] T 02aaa73cb73f4632a129ce70a9d4555f P 3515e5dbdb114e0dbe8c96b2af448ce3: No bootstrap required, opened a new log
I20260620 23:36:32.869700 8591 ts_tablet_manager.cc:1397] T 02aaa73cb73f4632a129ce70a9d4555f P 3515e5dbdb114e0dbe8c96b2af448ce3: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260620 23:36:32.869585 8594 consensus_queue.cc:260] T 02aaa73cb73f4632a129ce70a9d4555f P 86d9648f36394a698d81a8dc587f77c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40283 } } peers { permanent_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44049 } } peers { permanent_uuid: "86d9648f36394a698d81a8dc587f77c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40371 } }
I20260620 23:36:32.870676 8591 raft_consensus.cc:348] T 02aaa73cb73f4632a129ce70a9d4555f P 3515e5dbdb114e0dbe8c96b2af448ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40283 } } peers { permanent_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44049 } } peers { permanent_uuid: "86d9648f36394a698d81a8dc587f77c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40371 } }
I20260620 23:36:32.870961 8591 raft_consensus.cc:374] T 02aaa73cb73f4632a129ce70a9d4555f P 3515e5dbdb114e0dbe8c96b2af448ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:36:32.871043 8591 raft_consensus.cc:729] T 02aaa73cb73f4632a129ce70a9d4555f P 3515e5dbdb114e0dbe8c96b2af448ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3515e5dbdb114e0dbe8c96b2af448ce3, State: Initialized, Role: FOLLOWER
I20260620 23:36:32.871205 8594 ts_tablet_manager.cc:1428] T 02aaa73cb73f4632a129ce70a9d4555f P 86d9648f36394a698d81a8dc587f77c1: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 23:36:32.871326 8591 consensus_queue.cc:260] T 02aaa73cb73f4632a129ce70a9d4555f P 3515e5dbdb114e0dbe8c96b2af448ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40283 } } peers { permanent_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44049 } } peers { permanent_uuid: "86d9648f36394a698d81a8dc587f77c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40371 } }
I20260620 23:36:32.871841 8591 ts_tablet_manager.cc:1428] T 02aaa73cb73f4632a129ce70a9d4555f P 3515e5dbdb114e0dbe8c96b2af448ce3: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260620 23:36:32.873068 8590 tablet_bootstrap.cc:492] T 02aaa73cb73f4632a129ce70a9d4555f P e347312aaa774f81b9f0232b2f7f1d8d: No bootstrap required, opened a new log
I20260620 23:36:32.873268 8590 ts_tablet_manager.cc:1397] T 02aaa73cb73f4632a129ce70a9d4555f P e347312aaa774f81b9f0232b2f7f1d8d: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260620 23:36:32.874091 8590 raft_consensus.cc:348] T 02aaa73cb73f4632a129ce70a9d4555f P e347312aaa774f81b9f0232b2f7f1d8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40283 } } peers { permanent_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44049 } } peers { permanent_uuid: "86d9648f36394a698d81a8dc587f77c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40371 } }
I20260620 23:36:32.874392 8590 raft_consensus.cc:374] T 02aaa73cb73f4632a129ce70a9d4555f P e347312aaa774f81b9f0232b2f7f1d8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:36:32.874485 8590 raft_consensus.cc:729] T 02aaa73cb73f4632a129ce70a9d4555f P e347312aaa774f81b9f0232b2f7f1d8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e347312aaa774f81b9f0232b2f7f1d8d, State: Initialized, Role: FOLLOWER
I20260620 23:36:32.874758 8590 consensus_queue.cc:260] T 02aaa73cb73f4632a129ce70a9d4555f P e347312aaa774f81b9f0232b2f7f1d8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40283 } } peers { permanent_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44049 } } peers { permanent_uuid: "86d9648f36394a698d81a8dc587f77c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40371 } }
I20260620 23:36:32.875854 8590 ts_tablet_manager.cc:1428] T 02aaa73cb73f4632a129ce70a9d4555f P e347312aaa774f81b9f0232b2f7f1d8d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 23:36:33.041774 8555 raft_consensus.cc:482] T 02aaa73cb73f4632a129ce70a9d4555f P e347312aaa774f81b9f0232b2f7f1d8d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:36:33.042049 8555 raft_consensus.cc:504] T 02aaa73cb73f4632a129ce70a9d4555f P e347312aaa774f81b9f0232b2f7f1d8d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40283 } } peers { permanent_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44049 } } peers { permanent_uuid: "86d9648f36394a698d81a8dc587f77c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40371 } }
I20260620 23:36:33.042724 8555 leader_election.cc:288] T 02aaa73cb73f4632a129ce70a9d4555f P e347312aaa774f81b9f0232b2f7f1d8d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3515e5dbdb114e0dbe8c96b2af448ce3 (127.0.97.1:40283), 86d9648f36394a698d81a8dc587f77c1 (127.0.97.3:40371)
I20260620 23:36:33.043397 8160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02aaa73cb73f4632a129ce70a9d4555f" candidate_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" is_pre_election: true
I20260620 23:36:33.043469 8438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02aaa73cb73f4632a129ce70a9d4555f" candidate_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86d9648f36394a698d81a8dc587f77c1" is_pre_election: true
I20260620 23:36:33.043689 8160 raft_consensus.cc:2454] T 02aaa73cb73f4632a129ce70a9d4555f P 3515e5dbdb114e0dbe8c96b2af448ce3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e347312aaa774f81b9f0232b2f7f1d8d in term 0.
I20260620 23:36:33.043689 8438 raft_consensus.cc:2454] T 02aaa73cb73f4632a129ce70a9d4555f P 86d9648f36394a698d81a8dc587f77c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e347312aaa774f81b9f0232b2f7f1d8d in term 0.
I20260620 23:36:33.044155 8233 leader_election.cc:302] T 02aaa73cb73f4632a129ce70a9d4555f P e347312aaa774f81b9f0232b2f7f1d8d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3515e5dbdb114e0dbe8c96b2af448ce3, e347312aaa774f81b9f0232b2f7f1d8d; no voters:
I20260620 23:36:33.044482 8555 raft_consensus.cc:2781] T 02aaa73cb73f4632a129ce70a9d4555f P e347312aaa774f81b9f0232b2f7f1d8d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:36:33.044593 8555 raft_consensus.cc:482] T 02aaa73cb73f4632a129ce70a9d4555f P e347312aaa774f81b9f0232b2f7f1d8d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:36:33.044761 8555 raft_consensus.cc:3037] T 02aaa73cb73f4632a129ce70a9d4555f P e347312aaa774f81b9f0232b2f7f1d8d [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:36:33.047607 8555 raft_consensus.cc:504] T 02aaa73cb73f4632a129ce70a9d4555f P e347312aaa774f81b9f0232b2f7f1d8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40283 } } peers { permanent_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44049 } } peers { permanent_uuid: "86d9648f36394a698d81a8dc587f77c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40371 } }
I20260620 23:36:33.048225 8555 leader_election.cc:288] T 02aaa73cb73f4632a129ce70a9d4555f P e347312aaa774f81b9f0232b2f7f1d8d [CANDIDATE]: Term 1 election: Requested vote from peers 3515e5dbdb114e0dbe8c96b2af448ce3 (127.0.97.1:40283), 86d9648f36394a698d81a8dc587f77c1 (127.0.97.3:40371)
I20260620 23:36:33.048760 8160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02aaa73cb73f4632a129ce70a9d4555f" candidate_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3"
I20260620 23:36:33.048743 8438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02aaa73cb73f4632a129ce70a9d4555f" candidate_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86d9648f36394a698d81a8dc587f77c1"
I20260620 23:36:33.048974 8160 raft_consensus.cc:3037] T 02aaa73cb73f4632a129ce70a9d4555f P 3515e5dbdb114e0dbe8c96b2af448ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:36:33.048982 8438 raft_consensus.cc:3037] T 02aaa73cb73f4632a129ce70a9d4555f P 86d9648f36394a698d81a8dc587f77c1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:36:33.051151 8160 raft_consensus.cc:2454] T 02aaa73cb73f4632a129ce70a9d4555f P 3515e5dbdb114e0dbe8c96b2af448ce3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e347312aaa774f81b9f0232b2f7f1d8d in term 1.
I20260620 23:36:33.051595 8233 leader_election.cc:302] T 02aaa73cb73f4632a129ce70a9d4555f P e347312aaa774f81b9f0232b2f7f1d8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3515e5dbdb114e0dbe8c96b2af448ce3, e347312aaa774f81b9f0232b2f7f1d8d; no voters:
I20260620 23:36:33.051867 8555 raft_consensus.cc:2781] T 02aaa73cb73f4632a129ce70a9d4555f P e347312aaa774f81b9f0232b2f7f1d8d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:36:33.052050 8555 raft_consensus.cc:686] T 02aaa73cb73f4632a129ce70a9d4555f P e347312aaa774f81b9f0232b2f7f1d8d [term 1 LEADER]: Becoming Leader. State: Replica: e347312aaa774f81b9f0232b2f7f1d8d, State: Running, Role: LEADER
I20260620 23:36:33.052066 8438 raft_consensus.cc:2454] T 02aaa73cb73f4632a129ce70a9d4555f P 86d9648f36394a698d81a8dc587f77c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e347312aaa774f81b9f0232b2f7f1d8d in term 1.
I20260620 23:36:33.052316 8555 consensus_queue.cc:237] T 02aaa73cb73f4632a129ce70a9d4555f P e347312aaa774f81b9f0232b2f7f1d8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40283 } } peers { permanent_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44049 } } peers { permanent_uuid: "86d9648f36394a698d81a8dc587f77c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40371 } }
I20260620 23:36:33.055226 8003 catalog_manager.cc:5495] T 02aaa73cb73f4632a129ce70a9d4555f P e347312aaa774f81b9f0232b2f7f1d8d reported cstate change: term changed from 0 to 1, leader changed from <none> to e347312aaa774f81b9f0232b2f7f1d8d (127.0.97.2). New cstate: current_term: 1 leader_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e347312aaa774f81b9f0232b2f7f1d8d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "86d9648f36394a698d81a8dc587f77c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40371 } health_report { overall_health: UNKNOWN } } }
I20260620 23:36:33.089084 8512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:33.083334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34609 (local address 127.0.97.2:44049)
0620 23:36:33.083554 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:33.083562 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:33.084028 (+ 466us) server_negotiation.cc:407] Connection header received
0620 23:36:33.084233 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:33.084237 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:33.084313 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:33.084398 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:36:33.084881 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:33.085667 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:33.087153 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:33.087571 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:33.087714 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:33.088148 (+ 434us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:36:33.088266 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:36:33.088391 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:33.088548 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 23:36:33.088636 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":60}
I20260620 23:36:33.089107 8606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:33.083233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44049 (local address 127.0.97.2:34609)
0620 23:36:33.083925 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:33.083943 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:33.084110 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:33.084417 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:33.084421 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:33.084431 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:36:33.084733 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:33.084742 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:33.085815 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:33.085821 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:33.087011 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:33.087032 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:33.087788 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:33.087795 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:33.087940 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:33.087974 (+ 34us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:36:33.088391 (+ 417us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:36:33.088405 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 23:36:33.088525 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 23:36:33.088671 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":371,"threads_started":1}
I20260620 23:36:33.098276 8160 raft_consensus.cc:1261] T 02aaa73cb73f4632a129ce70a9d4555f P 3515e5dbdb114e0dbe8c96b2af448ce3 [term 1 FOLLOWER]: Refusing update from remote peer e347312aaa774f81b9f0232b2f7f1d8d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:36:33.098502 8438 raft_consensus.cc:1261] T 02aaa73cb73f4632a129ce70a9d4555f P 86d9648f36394a698d81a8dc587f77c1 [term 1 FOLLOWER]: Refusing update from remote peer e347312aaa774f81b9f0232b2f7f1d8d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:36:33.098851 8555 consensus_queue.cc:1035] T 02aaa73cb73f4632a129ce70a9d4555f P e347312aaa774f81b9f0232b2f7f1d8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3515e5dbdb114e0dbe8c96b2af448ce3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:36:33.099227 8520 consensus_queue.cc:1035] T 02aaa73cb73f4632a129ce70a9d4555f P e347312aaa774f81b9f0232b2f7f1d8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "86d9648f36394a698d81a8dc587f77c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:36:33.169135 8003 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:36:33.169428 8003 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:36:33.172513 8003 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 09f79b3ac58944c1be69f25d4e293a77: Sending DeleteTablet for 3 replicas of tablet 02aaa73cb73f4632a129ce70a9d4555f
I20260620 23:36:33.173583 8139 tablet_service.cc:1514] Processing DeleteTablet for tablet 02aaa73cb73f4632a129ce70a9d4555f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:36:33 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41472
I20260620 23:36:33.173580 8418 tablet_service.cc:1514] Processing DeleteTablet for tablet 02aaa73cb73f4632a129ce70a9d4555f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:36:33 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36820
I20260620 23:36:33.173905 8574 tablet_replica.cc:330] T 02aaa73cb73f4632a129ce70a9d4555f P 86d9648f36394a698d81a8dc587f77c1: stopping tablet replica
I20260620 23:36:33.174106 8574 raft_consensus.cc:2229] T 02aaa73cb73f4632a129ce70a9d4555f P 86d9648f36394a698d81a8dc587f77c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:36:33.174341 8574 raft_consensus.cc:2258] T 02aaa73cb73f4632a129ce70a9d4555f P 86d9648f36394a698d81a8dc587f77c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:36:33.174657 8576 tablet_replica.cc:330] T 02aaa73cb73f4632a129ce70a9d4555f P 3515e5dbdb114e0dbe8c96b2af448ce3: stopping tablet replica
I20260620 23:36:33.174878 8576 raft_consensus.cc:2229] T 02aaa73cb73f4632a129ce70a9d4555f P 3515e5dbdb114e0dbe8c96b2af448ce3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:36:33.175123 8576 raft_consensus.cc:2258] T 02aaa73cb73f4632a129ce70a9d4555f P 3515e5dbdb114e0dbe8c96b2af448ce3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:36:33.175163 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 8074
I20260620 23:36:33.175443 8574 ts_tablet_manager.cc:1905] T 02aaa73cb73f4632a129ce70a9d4555f P 86d9648f36394a698d81a8dc587f77c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:36:33.176349 8279 tablet_service.cc:1514] Processing DeleteTablet for tablet 02aaa73cb73f4632a129ce70a9d4555f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:36:33 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42826
I20260620 23:36:33.176666 8575 tablet_replica.cc:330] T 02aaa73cb73f4632a129ce70a9d4555f P e347312aaa774f81b9f0232b2f7f1d8d: stopping tablet replica
I20260620 23:36:33.176837 8575 raft_consensus.cc:2229] T 02aaa73cb73f4632a129ce70a9d4555f P e347312aaa774f81b9f0232b2f7f1d8d [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:36:33.177150 8575 raft_consensus.cc:2258] T 02aaa73cb73f4632a129ce70a9d4555f P e347312aaa774f81b9f0232b2f7f1d8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:36:33.177273 8576 ts_tablet_manager.cc:1905] T 02aaa73cb73f4632a129ce70a9d4555f P 3515e5dbdb114e0dbe8c96b2af448ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:36:33.181980 8575 ts_tablet_manager.cc:1905] T 02aaa73cb73f4632a129ce70a9d4555f P e347312aaa774f81b9f0232b2f7f1d8d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:36:33.183373 8574 ts_tablet_manager.cc:1918] T 02aaa73cb73f4632a129ce70a9d4555f P 86d9648f36394a698d81a8dc587f77c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:36:33.183694 8574 log.cc:1196] T 02aaa73cb73f4632a129ce70a9d4555f P 86d9648f36394a698d81a8dc587f77c1: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/ts-2/wal/wals/02aaa73cb73f4632a129ce70a9d4555f
I20260620 23:36:33.184243 8574 ts_tablet_manager.cc:1939] T 02aaa73cb73f4632a129ce70a9d4555f P 86d9648f36394a698d81a8dc587f77c1: Deleting consensus metadata
I20260620 23:36:33.187482 7990 catalog_manager.cc:4841] TS 86d9648f36394a698d81a8dc587f77c1 (127.0.97.3:40371): tablet 02aaa73cb73f4632a129ce70a9d4555f (table test-table [id=0cc7083b34b844ffad9528e02ca8ce15]) successfully deleted
I20260620 23:36:33.193419 8576 ts_tablet_manager.cc:1918] T 02aaa73cb73f4632a129ce70a9d4555f P 3515e5dbdb114e0dbe8c96b2af448ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:36:33.193638 8576 log.cc:1196] T 02aaa73cb73f4632a129ce70a9d4555f P 3515e5dbdb114e0dbe8c96b2af448ce3: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/ts-0/wal/wals/02aaa73cb73f4632a129ce70a9d4555f
I20260620 23:36:33.194149 8576 ts_tablet_manager.cc:1939] T 02aaa73cb73f4632a129ce70a9d4555f P 3515e5dbdb114e0dbe8c96b2af448ce3: Deleting consensus metadata
I20260620 23:36:33.194604 8575 ts_tablet_manager.cc:1918] T 02aaa73cb73f4632a129ce70a9d4555f P e347312aaa774f81b9f0232b2f7f1d8d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:36:33.194850 8575 log.cc:1196] T 02aaa73cb73f4632a129ce70a9d4555f P e347312aaa774f81b9f0232b2f7f1d8d: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998525501163-388-0/minicluster-data/ts-1/wal/wals/02aaa73cb73f4632a129ce70a9d4555f
I20260620 23:36:33.195360 8575 ts_tablet_manager.cc:1939] T 02aaa73cb73f4632a129ce70a9d4555f P e347312aaa774f81b9f0232b2f7f1d8d: Deleting consensus metadata
I20260620 23:36:33.195806 7990 catalog_manager.cc:4841] TS 3515e5dbdb114e0dbe8c96b2af448ce3 (127.0.97.1:40283): tablet 02aaa73cb73f4632a129ce70a9d4555f (table test-table [id=0cc7083b34b844ffad9528e02ca8ce15]) successfully deleted
I20260620 23:36:33.196733 7990 catalog_manager.cc:4841] TS e347312aaa774f81b9f0232b2f7f1d8d (127.0.97.2:44049): tablet 02aaa73cb73f4632a129ce70a9d4555f (table test-table [id=0cc7083b34b844ffad9528e02ca8ce15]) successfully deleted
Jun 20 23:36:33 dist-test-slave-tcgx krb5kdc[7958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260620 23:36:33.214366 8611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:33.178477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41490 (local address 127.0.97.1:40283)
0620 23:36:33.190058 (+ 11581us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:33.190065 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:33.197119 (+ 7054us) server_negotiation.cc:407] Connection header received
0620 23:36:33.197308 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:33.197312 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:33.197376 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:33.197475 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:33.199147 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:33.199879 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:33.201649 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:33.202062 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:33.202154 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:33.207329 (+ 5175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:33.207353 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:33.207374 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:33.207429 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:33.211361 (+ 3932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:33.212127 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:33.212138 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:33.212142 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:33.212314 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:33.212682 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:33.212687 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:33.212690 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:33.213160 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:33.213501 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:33.213687 (+ 186us) server_negotiation.cc:299] Negotiation successful
0620 23:36:33.213952 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11416,"thread_start_us":11248,"threads_started":1}
I20260620 23:36:33.215183 8205 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:36:33.340848 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 8074
I20260620 23:36:33.364864 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 8214
Jun 20 23:36:33 dist-test-slave-tcgx krb5kdc[7958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260620 23:36:33.380805 8512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:33.365579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42842 (local address 127.0.97.2:44049)
0620 23:36:33.365779 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:33.365787 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:33.365851 (+ 64us) server_negotiation.cc:407] Connection header received
0620 23:36:33.366050 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:33.366055 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:33.366139 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:33.366262 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:33.367559 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:33.368302 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:33.369585 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:33.370039 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:33.370178 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:33.374677 (+ 4499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:33.374703 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:33.374718 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:33.374758 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:33.377801 (+ 3043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:33.378579 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:33.378589 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:33.378591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:33.378727 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:33.379192 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:33.379196 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:33.379198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:33.379564 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:33.379876 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:33.380218 (+ 342us) server_negotiation.cc:299] Negotiation successful
0620 23:36:33.380464 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 23:36:33.381486 8345 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:36:33.530537 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 8214
I20260620 23:36:33.556466 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 8354
Jun 20 23:36:33 dist-test-slave-tcgx krb5kdc[7958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260620 23:36:33.573071 8614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:33.557316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36826 (local address 127.0.97.3:40371)
0620 23:36:33.557863 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:33.557867 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:33.557884 (+ 17us) server_negotiation.cc:407] Connection header received
0620 23:36:33.557945 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:33.557949 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:33.558009 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:33.558101 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:33.559585 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:33.560366 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:33.561628 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:33.562079 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:33.562190 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:33.566898 (+ 4708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:33.566914 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:33.566932 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:33.567046 (+ 114us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:33.569909 (+ 2863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:33.570617 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:33.570625 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:33.570627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:33.570777 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:33.571461 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:33.571468 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:33.571471 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:33.571920 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:33.572183 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:33.572459 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 23:36:33.572663 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":317,"threads_started":1}
I20260620 23:36:33.573823 8484 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:36:33.673564 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 8354
I20260620 23:36:33.691290 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 7971
I20260620 23:36:33.692510 8043 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:36:33.809543 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 7971
2026-06-20T23:36:33Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5934 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781998525501163-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:36:33 dist-test-slave-tcgx krb5kdc[8619](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:36:33 dist-test-slave-tcgx krb5kdc[8619](info): set up 2 sockets
Jun 20 23:36:33 dist-test-slave-tcgx krb5kdc[8619](info): commencing operation
krb5kdc: starting...
W20260620 23:36:34.932101 388 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.003s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781998525501163-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781998525501163-388-0/krb5kdc/test-user.keytab.
Jun 20 23:36:35 dist-test-slave-tcgx krb5kdc[8619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998595, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:36:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:36:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 23:36:35.164391 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781998525501163-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781998525501163-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781998525501163-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42711
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42711
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:36:35.508697 8632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:36:35.509079 8632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:36:35.509256 8632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:36:35.524544 8632 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:36:35.524801 8632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:36:35.524937 8632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:36:35.525101 8632 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:36:35.525269 8632 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:36:35.543823 8632 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781998525501163-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781998525501163-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42711
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781998525501163-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:36:35.545454 8632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:36:35.547006 8632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:36:35.560500 8637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:35.560415 8638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:35.560472 8640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:36:35.561511 8632 server_base.cc:1034] running on GCE node
I20260620 23:36:35.562484 8632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:36:35.564333 8632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:36:35.565606 8632 hybrid_clock.cc:648] HybridClock initialized: now 1781998595565540 us; error 56 us; skew 500 ppm
I20260620 23:36:35.566058 8632 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T23:36:35Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1829 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:36:35 dist-test-slave-tcgx krb5kdc[8646](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:36:35 dist-test-slave-tcgx krb5kdc[8646](info): set up 2 sockets
Jun 20 23:36:35 dist-test-slave-tcgx krb5kdc[8646](info): commencing operation
krb5kdc: starting...
W20260620 23:36:36.751972 388 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.006s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/krb5kdc/test-user.keytab.
Jun 20 23:36:36 dist-test-slave-tcgx krb5kdc[8646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:36:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:36:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 23:36:36.979032 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41309
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41309
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:36:37.313308 8659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:36:37.313637 8659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:36:37.313740 8659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:36:37.321327 8659 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:36:37.321456 8659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:36:37.321503 8659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:36:37.321552 8659 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:36:37.321597 8659 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:36:37.331816 8659 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41309
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:36:37.333168 8659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:36:37.334465 8659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:36:37.346624 8665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:37.346645 8664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:36:37.347299 8659 server_base.cc:1034] running on GCE node
W20260620 23:36:37.347434 8667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:36:37.348282 8659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:36:37.349822 8659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:36:37.351070 8659 hybrid_clock.cc:648] HybridClock initialized: now 1781998597351004 us; error 62 us; skew 500 ppm
I20260620 23:36:37.351457 8659 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:36:37 dist-test-slave-tcgx krb5kdc[8646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998597, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:36:37.357853 8659 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 23:36:37.359565 8659 webserver.cc:460] Webserver started at http://127.0.97.62:34739/ using document root <none> and password file <none>
I20260620 23:36:37.360272 8659 fs_manager.cc:359] Metadata directory not provided
I20260620 23:36:37.360379 8659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:36:37.360668 8659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:36:37.363902 8659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/master-0/data/instance:
uuid: "06503c2a6e2a48e4844cd168b6e99443"
format_stamp: "Formatted at 2026-06-20 23:36:37 on dist-test-slave-tcgx"
server_key: "b38d7731aeda9c7a7119e2f3d2b65ce4"
server_key_iv: "19655ace7ea52209d32a2b122e43ee30"
server_key_version: "encryptionkey@0"
I20260620 23:36:37.364753 8659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/master-0/wal/instance:
uuid: "06503c2a6e2a48e4844cd168b6e99443"
format_stamp: "Formatted at 2026-06-20 23:36:37 on dist-test-slave-tcgx"
server_key: "b38d7731aeda9c7a7119e2f3d2b65ce4"
server_key_iv: "19655ace7ea52209d32a2b122e43ee30"
server_key_version: "encryptionkey@0"
I20260620 23:36:37.370522 8659 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 23:36:37.374347 8674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:37.375998 8659 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 23:36:37.376205 8659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/master-0/wal
uuid: "06503c2a6e2a48e4844cd168b6e99443"
format_stamp: "Formatted at 2026-06-20 23:36:37 on dist-test-slave-tcgx"
server_key: "b38d7731aeda9c7a7119e2f3d2b65ce4"
server_key_iv: "19655ace7ea52209d32a2b122e43ee30"
server_key_version: "encryptionkey@0"
I20260620 23:36:37.376443 8659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:36:37.390148 8659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:36:37.394695 8659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:36:37.395056 8659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:36:37.414462 8659 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41309
I20260620 23:36:37.414492 8725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41309 every 8 connection(s)
I20260620 23:36:37.416059 8659 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:36:37.419916 8726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:36:37.421728 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 8659
I20260620 23:36:37.421957 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/master-0/wal/instance
I20260620 23:36:37.422784 388 external_mini_cluster.cc:1428] Setting key 99a75d1b84f0b6505b33c8d9f89c76ce
I20260620 23:36:37.433212 8726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06503c2a6e2a48e4844cd168b6e99443: Bootstrap starting.
I20260620 23:36:37.437479 8726 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06503c2a6e2a48e4844cd168b6e99443: Neither blocks nor log segments found. Creating new log.
Jun 20 23:36:37 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:36:37.438644 8726 log.cc:824] T 00000000000000000000000000000000 P 06503c2a6e2a48e4844cd168b6e99443: Log is configured to *not* fsync() on all Append() calls
I20260620 23:36:37.441759 8726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06503c2a6e2a48e4844cd168b6e99443: No bootstrap required, opened a new log
I20260620 23:36:37.446818 8729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:37.425036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49492 (local address 127.0.97.62:41309)
0620 23:36:37.425883 (+ 847us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:37.425898 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:37.425946 (+ 48us) server_negotiation.cc:407] Connection header received
0620 23:36:37.427728 (+ 1782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:37.427753 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:37.428169 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:37.428500 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:37.430294 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:37.431739 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:37.433502 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:37.434217 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:37.434354 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:37.439221 (+ 4867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:37.439230 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:37.439246 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:37.439286 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:37.442469 (+ 3183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:37.443281 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:37.443288 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:37.443298 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:37.443480 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:37.443888 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:37.443893 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:37.443896 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:37.444477 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:37.444824 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:37.445296 (+ 472us) server_negotiation.cc:299] Negotiation successful
0620 23:36:37.445738 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":406,"threads_started":1}
I20260620 23:36:37.450353 8726 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06503c2a6e2a48e4844cd168b6e99443 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06503c2a6e2a48e4844cd168b6e99443" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41309 } }
I20260620 23:36:37.450747 8726 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06503c2a6e2a48e4844cd168b6e99443 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:36:37.450857 8726 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06503c2a6e2a48e4844cd168b6e99443 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06503c2a6e2a48e4844cd168b6e99443, State: Initialized, Role: FOLLOWER
I20260620 23:36:37.451519 8726 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06503c2a6e2a48e4844cd168b6e99443 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06503c2a6e2a48e4844cd168b6e99443" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41309 } }
I20260620 23:36:37.451776 8726 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06503c2a6e2a48e4844cd168b6e99443 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:36:37.451913 8726 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06503c2a6e2a48e4844cd168b6e99443 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:36:37.452098 8726 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06503c2a6e2a48e4844cd168b6e99443 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:36:37.454282 8726 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06503c2a6e2a48e4844cd168b6e99443 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06503c2a6e2a48e4844cd168b6e99443" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41309 } }
I20260620 23:36:37.454979 8726 leader_election.cc:302] T 00000000000000000000000000000000 P 06503c2a6e2a48e4844cd168b6e99443 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06503c2a6e2a48e4844cd168b6e99443; no voters:
I20260620 23:36:37.455672 8726 leader_election.cc:288] T 00000000000000000000000000000000 P 06503c2a6e2a48e4844cd168b6e99443 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:36:37.455814 8731 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06503c2a6e2a48e4844cd168b6e99443 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:36:37.456498 8731 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06503c2a6e2a48e4844cd168b6e99443 [term 1 LEADER]: Becoming Leader. State: Replica: 06503c2a6e2a48e4844cd168b6e99443, State: Running, Role: LEADER
I20260620 23:36:37.457042 8731 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06503c2a6e2a48e4844cd168b6e99443 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06503c2a6e2a48e4844cd168b6e99443" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41309 } }
I20260620 23:36:37.457854 8726 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06503c2a6e2a48e4844cd168b6e99443 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:36:37.459921 8733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06503c2a6e2a48e4844cd168b6e99443 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06503c2a6e2a48e4844cd168b6e99443. Latest consensus state: current_term: 1 leader_uuid: "06503c2a6e2a48e4844cd168b6e99443" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06503c2a6e2a48e4844cd168b6e99443" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41309 } } }
I20260620 23:36:37.460489 8733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06503c2a6e2a48e4844cd168b6e99443 [sys.catalog]: This master's current role is: LEADER
I20260620 23:36:37.461177 8732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06503c2a6e2a48e4844cd168b6e99443 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06503c2a6e2a48e4844cd168b6e99443" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06503c2a6e2a48e4844cd168b6e99443" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41309 } } }
I20260620 23:36:37.463922 8732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06503c2a6e2a48e4844cd168b6e99443 [sys.catalog]: This master's current role is: LEADER
I20260620 23:36:37.464515 8739 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:36:37.470008 8739 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:36:37.479542 8739 catalog_manager.cc:1347] Generated new cluster ID: 5070943ba30b461197e08bd844a7c8e4
I20260620 23:36:37.479683 8739 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:36:37.498855 8739 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:36:37.500311 8739 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:36:37.509370 8739 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06503c2a6e2a48e4844cd168b6e99443: Generated new TSK 0
I20260620 23:36:37.510519 8739 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 23:36:37.571854 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41309
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:36:37.915844 8752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:36:37.916164 8752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:36:37.916345 8752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:36:37.923928 8752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:36:37.924060 8752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:36:37.924249 8752 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 23:36:37.934471 8752 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41309
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:36:37.935891 8752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:36:37.937307 8752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:36:37.951211 8757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:37.951201 8758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:37.951148 8760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:36:37.952591 8752 server_base.cc:1034] running on GCE node
I20260620 23:36:37.953317 8752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:36:37.954361 8752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:36:37.955610 8752 hybrid_clock.cc:648] HybridClock initialized: now 1781998597955546 us; error 58 us; skew 500 ppm
I20260620 23:36:37.955987 8752 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:36:37 dist-test-slave-tcgx krb5kdc[8646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998597, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:36:37.959322 8752 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 23:36:37.960954 8752 webserver.cc:460] Webserver started at http://127.0.97.1:34825/ using document root <none> and password file <none>
I20260620 23:36:37.961632 8752 fs_manager.cc:359] Metadata directory not provided
I20260620 23:36:37.961730 8752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:36:37.961988 8752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:36:37.965374 8752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-0/data/instance:
uuid: "e76e3b0545564ecca1ad59f5e124404d"
format_stamp: "Formatted at 2026-06-20 23:36:37 on dist-test-slave-tcgx"
server_key: "c6f49e48fa908889a5f933e190788853"
server_key_iv: "3fae12bf0a9a652104cb0fde09fda841"
server_key_version: "encryptionkey@0"
I20260620 23:36:37.966212 8752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-0/wal/instance:
uuid: "e76e3b0545564ecca1ad59f5e124404d"
format_stamp: "Formatted at 2026-06-20 23:36:37 on dist-test-slave-tcgx"
server_key: "c6f49e48fa908889a5f933e190788853"
server_key_iv: "3fae12bf0a9a652104cb0fde09fda841"
server_key_version: "encryptionkey@0"
I20260620 23:36:37.972262 8752 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260620 23:36:37.976101 8767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:37.977655 8752 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 23:36:37.977849 8752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-0/wal
uuid: "e76e3b0545564ecca1ad59f5e124404d"
format_stamp: "Formatted at 2026-06-20 23:36:37 on dist-test-slave-tcgx"
server_key: "c6f49e48fa908889a5f933e190788853"
server_key_iv: "3fae12bf0a9a652104cb0fde09fda841"
server_key_version: "encryptionkey@0"
I20260620 23:36:37.978073 8752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:36:38.001374 8752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:36:38.006145 8752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:36:38.006464 8752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:36:38.007609 8752 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:36:38.009011 8752 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:36:38.009088 8752 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:38.009272 8752 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:36:38.009361 8752 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:38.042850 8752 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44865
I20260620 23:36:38.043005 8879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44865 every 8 connection(s)
I20260620 23:36:38.044283 8752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
I20260620 23:36:38.045297 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 8752
I20260620 23:36:38.045527 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-0/wal/instance
I20260620 23:36:38.046342 388 external_mini_cluster.cc:1428] Setting key ecdeb462d0baa2a38fd319cbba52a279
Jun 20 23:36:38 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998597, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:36:38.068517 8882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:38.047593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:32921 (local address 127.0.97.62:41309)
0620 23:36:38.048106 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:38.048112 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:38.049221 (+ 1109us) server_negotiation.cc:407] Connection header received
0620 23:36:38.052214 (+ 2993us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:38.052219 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:38.052292 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:38.052388 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:38.054636 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:38.055444 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:38.057564 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:38.057955 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:38.058170 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:38.061797 (+ 3627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:38.061802 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:38.061805 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:38.061837 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:38.064929 (+ 3092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:38.065874 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:38.065883 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:38.065890 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:38.066058 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:38.066494 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:38.066502 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:38.066508 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:38.066927 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:38.067143 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:38.067834 (+ 691us) server_negotiation.cc:299] Negotiation successful
0620 23:36:38.068086 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":265,"threads_started":1}
I20260620 23:36:38.069367 8883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:38.048102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41309 (local address 127.0.97.1:32921)
0620 23:36:38.049032 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:38.049093 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:38.051886 (+ 2793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:38.052687 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:38.052702 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:38.053535 (+ 833us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:38.054406 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:38.054442 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:38.055609 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:38.055618 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:38.057370 (+ 1752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:38.057382 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:38.058274 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:38.058281 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:38.058516 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:38.059332 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:38.059356 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:38.061537 (+ 2181us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:36:38.065108 (+ 3571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:38.065118 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:38.065145 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:38.065670 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:38.066186 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:38.066189 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:38.066191 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:38.066377 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:38.067082 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:36:38.067088 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:36:38.067432 (+ 344us) client_negotiation.cc:769] Sending connection context
0620 23:36:38.067670 (+ 238us) client_negotiation.cc:240] Negotiation successful
0620 23:36:38.068125 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":385,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 23:36:38.071080 8880 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41309
I20260620 23:36:38.071411 8880 heartbeater.cc:464] Registering TS with master...
I20260620 23:36:38.072223 8880 heartbeater.cc:511] Master 127.0.97.62:41309 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260620 23:36:38.074605 8691 ts_manager.cc:194] Registered new tserver with Master: e76e3b0545564ecca1ad59f5e124404d (127.0.97.1:44865)
I20260620 23:36:38.076220 8691 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:32921
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 23:36:38.100096 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41309
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:36:38.438233 8886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:36:38.438603 8886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:36:38.438786 8886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:36:38.446494 8886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:36:38.446669 8886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:36:38.446909 8886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 23:36:38.457209 8886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41309
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:36:38.458758 8886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:36:38.460222 8886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:36:38.474113 8891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:38.474171 8892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:38.474387 8894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:36:38.475297 8886 server_base.cc:1034] running on GCE node
I20260620 23:36:38.475965 8886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:36:38.477002 8886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:36:38.478256 8886 hybrid_clock.cc:648] HybridClock initialized: now 1781998598478202 us; error 39 us; skew 500 ppm
I20260620 23:36:38.478614 8886 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:36:38 dist-test-slave-tcgx krb5kdc[8646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998598, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:36:38.482084 8886 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 23:36:38.483777 8886 webserver.cc:460] Webserver started at http://127.0.97.2:44045/ using document root <none> and password file <none>
I20260620 23:36:38.484535 8886 fs_manager.cc:359] Metadata directory not provided
I20260620 23:36:38.484699 8886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:36:38.485033 8886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:36:38.488303 8886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-1/data/instance:
uuid: "7dd27215fbe4481bbf8235001d0b8c79"
format_stamp: "Formatted at 2026-06-20 23:36:38 on dist-test-slave-tcgx"
server_key: "3c0658d6165b0e97e53b9be947277354"
server_key_iv: "f72a3a08ccff2300ad16343901b61ebf"
server_key_version: "encryptionkey@0"
I20260620 23:36:38.489189 8886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-1/wal/instance:
uuid: "7dd27215fbe4481bbf8235001d0b8c79"
format_stamp: "Formatted at 2026-06-20 23:36:38 on dist-test-slave-tcgx"
server_key: "3c0658d6165b0e97e53b9be947277354"
server_key_iv: "f72a3a08ccff2300ad16343901b61ebf"
server_key_version: "encryptionkey@0"
I20260620 23:36:38.495265 8886 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260620 23:36:38.499328 8901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:38.500983 8886 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:36:38.501207 8886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-1/wal
uuid: "7dd27215fbe4481bbf8235001d0b8c79"
format_stamp: "Formatted at 2026-06-20 23:36:38 on dist-test-slave-tcgx"
server_key: "3c0658d6165b0e97e53b9be947277354"
server_key_iv: "f72a3a08ccff2300ad16343901b61ebf"
server_key_version: "encryptionkey@0"
I20260620 23:36:38.501446 8886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:36:38.516279 8886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:36:38.520792 8886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:36:38.521106 8886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:36:38.522157 8886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:36:38.523627 8886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:36:38.523732 8886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:38.523900 8886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:36:38.523977 8886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:38.559059 8886 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40637
I20260620 23:36:38.559111 9013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40637 every 8 connection(s)
I20260620 23:36:38.560403 8886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
I20260620 23:36:38.563225 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 8886
I20260620 23:36:38.563474 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-1/wal/instance
I20260620 23:36:38.564039 388 external_mini_cluster.cc:1428] Setting key 162c72fc3c7124bdcf11b1c36d0d597e
Jun 20 23:36:38 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998598, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:36:38.582369 8882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:38.563232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38461 (local address 127.0.97.62:41309)
0620 23:36:38.563422 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:38.563428 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:38.565233 (+ 1805us) server_negotiation.cc:407] Connection header received
0620 23:36:38.567242 (+ 2009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:38.567247 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:38.567331 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:38.567457 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:38.569003 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:38.569742 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:38.571542 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:38.571991 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:38.572135 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:38.575892 (+ 3757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:38.575898 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:38.575901 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:38.575934 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:38.578836 (+ 2902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:38.579723 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:38.579736 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:38.579739 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:38.579862 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:38.580435 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:38.580439 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:38.580441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:38.580712 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:38.580876 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:38.581717 (+ 841us) server_negotiation.cc:299] Negotiation successful
0620 23:36:38.582006 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 23:36:38.583577 9016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:38.564041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41309 (local address 127.0.97.2:38461)
0620 23:36:38.565052 (+ 1011us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:38.565096 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:38.566919 (+ 1823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:38.567698 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:38.567707 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:38.568226 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:38.568815 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:38.568832 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:38.569909 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:38.569918 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:38.571395 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:38.571408 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:38.572231 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:38.572238 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:38.572480 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:38.573232 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:38.573256 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:38.575635 (+ 2379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:36:38.579077 (+ 3442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:38.579088 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:38.579112 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:38.579581 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:38.579995 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:38.580000 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:38.580004 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:38.580256 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:38.580849 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:36:38.580857 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:36:38.581253 (+ 396us) client_negotiation.cc:769] Sending connection context
0620 23:36:38.581620 (+ 367us) client_negotiation.cc:240] Negotiation successful
0620 23:36:38.582211 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":359,"threads_started":1}
I20260620 23:36:38.585170 9014 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41309
I20260620 23:36:38.585559 9014 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 23:36:38.586444 9014 heartbeater.cc:511] Master 127.0.97.62:41309 requested a full tablet report, sending...
I20260620 23:36:38.588104 8691 ts_manager.cc:194] Registered new tserver with Master: 7dd27215fbe4481bbf8235001d0b8c79 (127.0.97.2:40637)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260620 23:36:38.589131 8691 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:38461
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 23:36:38.614589 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41309
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:36:38.953001 9019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:36:38.953289 9019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:36:38.953439 9019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:36:38.960932 9019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:36:38.961053 9019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:36:38.961261 9019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 23:36:38.971500 9019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41309
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:36:38.972971 9019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:36:38.974331 9019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:36:38.987910 9024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:38.987955 9027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:38.987869 9025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:36:38.989173 9019 server_base.cc:1034] running on GCE node
I20260620 23:36:38.989916 9019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:36:38.990854 9019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:36:38.992105 9019 hybrid_clock.cc:648] HybridClock initialized: now 1781998598992037 us; error 63 us; skew 500 ppm
I20260620 23:36:38.992448 9019 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:36:38 dist-test-slave-tcgx krb5kdc[8646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998598, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:36:38.995868 9019 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 23:36:38.997339 9019 webserver.cc:460] Webserver started at http://127.0.97.3:43239/ using document root <none> and password file <none>
I20260620 23:36:38.998067 9019 fs_manager.cc:359] Metadata directory not provided
I20260620 23:36:38.998219 9019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:36:38.998553 9019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:36:39.001552 9019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-2/data/instance:
uuid: "e57494d0e2c047f385d1880190def5cd"
format_stamp: "Formatted at 2026-06-20 23:36:38 on dist-test-slave-tcgx"
server_key: "3f01503f0075f5614a0d64c2a46b12c6"
server_key_iv: "d55838f169047c2f3b810d90bf6ac6bc"
server_key_version: "encryptionkey@0"
I20260620 23:36:39.002369 9019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-2/wal/instance:
uuid: "e57494d0e2c047f385d1880190def5cd"
format_stamp: "Formatted at 2026-06-20 23:36:38 on dist-test-slave-tcgx"
server_key: "3f01503f0075f5614a0d64c2a46b12c6"
server_key_iv: "d55838f169047c2f3b810d90bf6ac6bc"
server_key_version: "encryptionkey@0"
I20260620 23:36:39.008132 9019 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 23:36:39.011771 9034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:39.013219 9019 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:36:39.013419 9019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-2/wal
uuid: "e57494d0e2c047f385d1880190def5cd"
format_stamp: "Formatted at 2026-06-20 23:36:38 on dist-test-slave-tcgx"
server_key: "3f01503f0075f5614a0d64c2a46b12c6"
server_key_iv: "d55838f169047c2f3b810d90bf6ac6bc"
server_key_version: "encryptionkey@0"
I20260620 23:36:39.013652 9019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:36:39.027196 9019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:36:39.031388 9019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:36:39.031694 9019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:36:39.032986 9019 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:36:39.034349 9019 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:36:39.034461 9019 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:39.034591 9019 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:36:39.034675 9019 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:39.075161 9019 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36495
I20260620 23:36:39.075232 9146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36495 every 8 connection(s)
I20260620 23:36:39.076489 9019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
I20260620 23:36:39.077096 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 9019
I20260620 23:36:39.077311 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-2/wal/instance
I20260620 23:36:39.077878 388 external_mini_cluster.cc:1428] Setting key 152b7a152a5fdf4b60274ee88e4138ec
I20260620 23:36:39.080956 8880 heartbeater.cc:503] Master 127.0.97.62:41309 was elected leader, sending a full tablet report...
Jun 20 23:36:39 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998598, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:36:39.098440 8882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:39.079578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39439 (local address 127.0.97.62:41309)
0620 23:36:39.079792 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:39.079800 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:39.081393 (+ 1593us) server_negotiation.cc:407] Connection header received
0620 23:36:39.083718 (+ 2325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:39.083723 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:39.083804 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:39.083924 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:39.085531 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:39.086298 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:39.088013 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:39.088406 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:39.088544 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:39.092263 (+ 3719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:39.092268 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:39.092271 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:39.092302 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:39.095309 (+ 3007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:39.096070 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:39.096080 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:39.096083 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:39.096206 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:39.096662 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:39.096672 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:39.096674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:39.096942 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:39.097105 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:39.097846 (+ 741us) server_negotiation.cc:299] Negotiation successful
0620 23:36:39.098085 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 23:36:39.099552 9149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:39.080346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41309 (local address 127.0.97.3:39439)
0620 23:36:39.081231 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:39.081300 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:39.083450 (+ 2150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:39.084174 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:39.084184 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:39.084773 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:39.085354 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:39.085372 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:39.086464 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:39.086470 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:39.087861 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:39.087874 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:39.088641 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:39.088647 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:39.088889 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:39.089580 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:39.089601 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:39.092045 (+ 2444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:36:39.095471 (+ 3426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:39.095481 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:39.095508 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:39.095935 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:39.096320 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:36:39.096325 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:36:39.096329 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:36:39.096539 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:36:39.097084 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:36:39.097091 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:36:39.097467 (+ 376us) client_negotiation.cc:769] Sending connection context
0620 23:36:39.097739 (+ 272us) client_negotiation.cc:240] Negotiation successful
0620 23:36:39.098295 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":809,"threads_started":1}
I20260620 23:36:39.100914 9147 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41309
I20260620 23:36:39.101219 9147 heartbeater.cc:464] Registering TS with master...
I20260620 23:36:39.101934 9147 heartbeater.cc:511] Master 127.0.97.62:41309 requested a full tablet report, sending...
I20260620 23:36:39.103531 8691 ts_manager.cc:194] Registered new tserver with Master: e57494d0e2c047f385d1880190def5cd (127.0.97.3:36495)
I20260620 23:36:39.104465 8691 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:39439
I20260620 23:36:39.106467 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/krb5kdc/oryx_127.0.97.1.keytab.
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/krb5kdc/oryx_127.0.97.2.keytab.
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260620 23:36:39.248215 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 8752
Jun 20 23:36:39 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 23:36:39.265753 9157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:39.249471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54178 (local address 127.0.97.1:44865)
0620 23:36:39.250305 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:39.250312 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:39.250340 (+ 28us) server_negotiation.cc:407] Connection header received
0620 23:36:39.250401 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:39.250407 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:39.250640 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:39.250793 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:39.252241 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:39.253242 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:39.254552 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:39.255216 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:39.255321 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:39.259694 (+ 4373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:39.259710 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:39.259726 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:39.259763 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:39.262627 (+ 2864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:39.263490 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:39.263499 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:39.263501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:39.263628 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:39.264048 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:39.264051 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:39.264054 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:39.264413 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:39.264755 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:39.265109 (+ 354us) server_negotiation.cc:299] Negotiation successful
0620 23:36:39.265387 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":501,"threads_started":1}
I20260620 23:36:39.267087 8875 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:36:39.350363 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 8752
I20260620 23:36:39.367153 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 8886
Jun 20 23:36:39 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 23:36:39.383630 9159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:39.368056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48326 (local address 127.0.97.2:40637)
0620 23:36:39.368557 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:39.368565 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:39.368590 (+ 25us) server_negotiation.cc:407] Connection header received
0620 23:36:39.368648 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:39.368654 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:39.368870 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:39.369002 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:39.370229 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:39.371318 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:39.372605 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:39.373253 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:39.373355 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:39.377587 (+ 4232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:39.377600 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:39.377615 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:39.377653 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:39.380823 (+ 3170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:39.381521 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:39.381528 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:39.381531 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:39.381682 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:39.382026 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:39.382029 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:39.382032 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:39.382487 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:39.382739 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:39.383041 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 23:36:39.383240 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":235,"threads_started":1}
I20260620 23:36:39.385000 9009 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:36:39.470909 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 8886
I20260620 23:36:39.487473 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 9019
Jun 20 23:36:39 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 23:36:39.504580 9161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:39.488540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41410 (local address 127.0.97.3:36495)
0620 23:36:39.489204 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:39.489211 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:39.489240 (+ 29us) server_negotiation.cc:407] Connection header received
0620 23:36:39.489302 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:39.489308 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:39.489551 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:39.489727 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:39.491054 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:39.492117 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:39.493453 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:39.494053 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:39.494203 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:39.498670 (+ 4467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:39.498684 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:39.498700 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:39.498748 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:39.501570 (+ 2822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:39.502301 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:39.502309 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:39.502311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:39.502440 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:39.502819 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:39.502826 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:39.502829 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:39.503317 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:39.503649 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:39.503955 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 23:36:39.504216 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":360,"threads_started":1}
I20260620 23:36:39.505889 9142 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:36:39.599536 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 9019
I20260620 23:36:39.615988 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 8659
I20260620 23:36:39.617121 8721 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:36:39.718791 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 8659
I20260620 23:36:39.740042 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41309
--webserver_interface=127.0.97.62
--webserver_port=34739
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41309
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:36:40.074676 9164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:36:40.075187 9164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:36:40.075371 9164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:36:40.082715 9164 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:36:40.082881 9164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:36:40.082986 9164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:36:40.083073 9164 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:36:40.083160 9164 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:36:40.093612 9164 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41309
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=34739
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:36:40.095088 9164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:36:40.096483 9164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:36:40.109125 9169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:40.109098 9172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:40.109125 9170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:36:40.110136 9164 server_base.cc:1034] running on GCE node
I20260620 23:36:40.111109 9164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:36:40.112708 9164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:36:40.113931 9164 hybrid_clock.cc:648] HybridClock initialized: now 1781998600113873 us; error 61 us; skew 500 ppm
I20260620 23:36:40.114331 9164 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:36:40 dist-test-slave-tcgx krb5kdc[8646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998600, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:36:40.117720 9164 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 23:36:40.119390 9164 webserver.cc:460] Webserver started at http://127.0.97.62:34739/ using document root <none> and password file <none>
I20260620 23:36:40.120100 9164 fs_manager.cc:359] Metadata directory not provided
I20260620 23:36:40.120218 9164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:36:40.125873 9164 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260620 23:36:40.128963 9179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:40.130831 9164 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:36:40.131073 9164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/master-0/wal
uuid: "06503c2a6e2a48e4844cd168b6e99443"
format_stamp: "Formatted at 2026-06-20 23:36:37 on dist-test-slave-tcgx"
server_key: "b38d7731aeda9c7a7119e2f3d2b65ce4"
server_key_iv: "19655ace7ea52209d32a2b122e43ee30"
server_key_version: "encryptionkey@0"
I20260620 23:36:40.132165 9164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:36:40.149086 9164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:36:40.153501 9164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:36:40.153798 9164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:36:40.172921 9164 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41309
I20260620 23:36:40.172957 9230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41309 every 8 connection(s)
I20260620 23:36:40.174512 9164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:36:40.181607 9231 sys_catalog.cc:263] Verifying existing consensus state
I20260620 23:36:40.182569 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 9164
I20260620 23:36:40.183413 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:44865
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:41309
--webserver_port=34825
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.1
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/krb5kdc/oryx_127.0.97.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:36:40.185220 9231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06503c2a6e2a48e4844cd168b6e99443: Bootstrap starting.
I20260620 23:36:40.204759 9231 log.cc:824] T 00000000000000000000000000000000 P 06503c2a6e2a48e4844cd168b6e99443: Log is configured to *not* fsync() on all Append() calls
I20260620 23:36:40.214059 9231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06503c2a6e2a48e4844cd168b6e99443: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:36:40.214627 9231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06503c2a6e2a48e4844cd168b6e99443: Bootstrap complete.
I20260620 23:36:40.223912 9231 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06503c2a6e2a48e4844cd168b6e99443 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06503c2a6e2a48e4844cd168b6e99443" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41309 } }
I20260620 23:36:40.224793 9231 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06503c2a6e2a48e4844cd168b6e99443 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06503c2a6e2a48e4844cd168b6e99443, State: Initialized, Role: FOLLOWER
I20260620 23:36:40.225473 9231 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06503c2a6e2a48e4844cd168b6e99443 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06503c2a6e2a48e4844cd168b6e99443" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41309 } }
I20260620 23:36:40.225723 9231 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06503c2a6e2a48e4844cd168b6e99443 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:36:40.225854 9231 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06503c2a6e2a48e4844cd168b6e99443 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:36:40.226046 9231 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06503c2a6e2a48e4844cd168b6e99443 [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:36:40.229821 9231 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06503c2a6e2a48e4844cd168b6e99443 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06503c2a6e2a48e4844cd168b6e99443" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41309 } }
I20260620 23:36:40.230319 9231 leader_election.cc:302] T 00000000000000000000000000000000 P 06503c2a6e2a48e4844cd168b6e99443 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06503c2a6e2a48e4844cd168b6e99443; no voters:
I20260620 23:36:40.231027 9231 leader_election.cc:288] T 00000000000000000000000000000000 P 06503c2a6e2a48e4844cd168b6e99443 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 23:36:40.231137 9235 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06503c2a6e2a48e4844cd168b6e99443 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 23:36:40.232553 9235 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06503c2a6e2a48e4844cd168b6e99443 [term 2 LEADER]: Becoming Leader. State: Replica: 06503c2a6e2a48e4844cd168b6e99443, State: Running, Role: LEADER
I20260620 23:36:40.233126 9235 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06503c2a6e2a48e4844cd168b6e99443 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06503c2a6e2a48e4844cd168b6e99443" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41309 } }
I20260620 23:36:40.233414 9231 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06503c2a6e2a48e4844cd168b6e99443 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:36:40.235751 9237 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06503c2a6e2a48e4844cd168b6e99443 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06503c2a6e2a48e4844cd168b6e99443. Latest consensus state: current_term: 2 leader_uuid: "06503c2a6e2a48e4844cd168b6e99443" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06503c2a6e2a48e4844cd168b6e99443" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41309 } } }
I20260620 23:36:40.236205 9237 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06503c2a6e2a48e4844cd168b6e99443 [sys.catalog]: This master's current role is: LEADER
I20260620 23:36:40.236581 9236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06503c2a6e2a48e4844cd168b6e99443 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "06503c2a6e2a48e4844cd168b6e99443" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06503c2a6e2a48e4844cd168b6e99443" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41309 } } }
I20260620 23:36:40.237017 9236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06503c2a6e2a48e4844cd168b6e99443 [sys.catalog]: This master's current role is: LEADER
I20260620 23:36:40.247716 9244 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:36:40.253844 9244 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:36:40.255342 9244 catalog_manager.cc:1259] Loaded cluster ID: 5070943ba30b461197e08bd844a7c8e4
I20260620 23:36:40.255461 9244 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:36:40.263545 9244 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:36:40.265859 9244 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 06503c2a6e2a48e4844cd168b6e99443: Loaded TSK: 0
I20260620 23:36:40.268131 9244 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 23:36:40.562711 9233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:36:40.563151 9233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:36:40.563323 9233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:36:40.571075 9233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:36:40.571225 9233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:36:40.571415 9233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 23:36:40.581727 9233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:44865
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=34825
--tserver_master_addrs=127.0.97.62:41309
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:36:40.583325 9233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:36:40.584690 9233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:36:40.599714 9258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:40.599691 9257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:36:40.600656 9233 server_base.cc:1034] running on GCE node
W20260620 23:36:40.601044 9260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:36:40.601732 9233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:36:40.602780 9233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:36:40.604054 9233 hybrid_clock.cc:648] HybridClock initialized: now 1781998600603975 us; error 68 us; skew 500 ppm
I20260620 23:36:40.604425 9233 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:36:40 dist-test-slave-tcgx krb5kdc[8646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998600, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:36:40.607812 9233 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260620 23:36:40.609537 9233 webserver.cc:460] Webserver started at http://127.0.97.1:34825/ using document root <none> and password file <none>
I20260620 23:36:40.610291 9233 fs_manager.cc:359] Metadata directory not provided
I20260620 23:36:40.610446 9233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:36:40.616243 9233 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.002s
I20260620 23:36:40.619236 9267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:40.620757 9233 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 23:36:40.620935 9233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-0/wal
uuid: "e76e3b0545564ecca1ad59f5e124404d"
format_stamp: "Formatted at 2026-06-20 23:36:37 on dist-test-slave-tcgx"
server_key: "c6f49e48fa908889a5f933e190788853"
server_key_iv: "3fae12bf0a9a652104cb0fde09fda841"
server_key_version: "encryptionkey@0"
I20260620 23:36:40.621918 9233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:36:40.647243 9233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:36:40.652230 9233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:36:40.652519 9233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:36:40.653548 9233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:36:40.654907 9233 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:36:40.655092 9233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:40.655221 9233 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:36:40.655308 9233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:40.686734 9233 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44865
I20260620 23:36:40.686766 9379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44865 every 8 connection(s)
I20260620 23:36:40.688248 9233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
I20260620 23:36:40.697378 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 9233
I20260620 23:36:40.698032 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:40637
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:41309
--webserver_port=44045
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.2
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/krb5kdc/oryx_127.0.97.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:36:40 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:36:40 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:36:40.707729 9383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:40.692164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41309 (local address 127.0.97.1:36461)
0620 23:36:40.692846 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:40.692891 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:40.694641 (+ 1750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:40.696139 (+ 1498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:40.696152 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:40.696784 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:40.697346 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:40.697363 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:40.699198 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:40.699202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:40.700441 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:40.700453 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:40.701555 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:40.701562 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:40.701846 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:40.702570 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:40.702594 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:40.706323 (+ 3729us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41309: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":301,"threads_started":1}
W20260620 23:36:40.707764 9382 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:36:40.691388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36461 (local address 127.0.97.62:41309)
0620 23:36:40.692009 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:40.692021 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:40.693044 (+ 1023us) server_negotiation.cc:407] Connection header received
0620 23:36:40.695249 (+ 2205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:40.695263 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:40.695620 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:40.695992 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:40.697583 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:40.699029 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:40.700609 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:40.701305 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:40.701531 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:40.706744 (+ 5213us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:36461: BlockingRecv error: recv got EOF from 127.0.97.1:36461 (error 108)
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":909,"threads_started":1}
W20260620 23:36:40.708423 9380 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:41309 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:41309: Client connection negotiation failed: client connection to 127.0.97.62:41309: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Jun 20 23:36:40 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:36:40 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 23:36:40.717338 9382 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:36:40.709179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46879 (local address 127.0.97.62:41309)
0620 23:36:40.709340 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:40.709346 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:40.709366 (+ 20us) server_negotiation.cc:407] Connection header received
0620 23:36:40.709567 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:40.709574 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:40.709640 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:40.709737 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:40.710416 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:40.711202 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:40.712403 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:40.712802 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:40.712935 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:40.717171 (+ 4236us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:46879: BlockingRecv error: recv got EOF from 127.0.97.1:46879 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 23:36:40.717367 9383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:40.709086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41309 (local address 127.0.97.1:46879)
0620 23:36:40.709268 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:40.709287 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:40.709401 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:40.709799 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:40.709803 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:40.710141 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:40.710275 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:40.710283 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:40.711338 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:40.711343 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:40.712256 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:40.712268 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:40.712947 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:40.712953 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:40.713134 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:40.713737 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:40.713754 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:40.716985 (+ 3231us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41309: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":60}
Jun 20 23:36:40 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:36:40 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:36:40.726624 9383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:40.718136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41309 (local address 127.0.97.1:32937)
0620 23:36:40.718329 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:40.718360 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:40.718471 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:40.718896 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:40.718899 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:40.719224 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:40.719366 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:40.719374 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:40.720915 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:40.720919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:40.721826 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:40.721836 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:40.722761 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:40.722767 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:40.722898 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:40.723555 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:40.723573 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:40.726343 (+ 2770us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41309: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":62}
W20260620 23:36:40.726894 9382 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:36:40.718316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:32937 (local address 127.0.97.62:41309)
0620 23:36:40.718522 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:40.718539 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:40.718563 (+ 24us) server_negotiation.cc:407] Connection header received
0620 23:36:40.718634 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:40.718642 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:40.718747 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:40.718863 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:40.719517 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:40.720778 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:40.721955 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:40.722584 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:40.722724 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:40.726621 (+ 3897us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:32937: BlockingRecv error: recv got EOF from 127.0.97.1:32937 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260620 23:36:40.727047 9380 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 23:36:41.039507 9384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:36:41.039811 9384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:36:41.039907 9384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:36:41.048056 9384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:36:41.048182 9384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:36:41.048297 9384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 23:36:41.058255 9384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:40637
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=44045
--tserver_master_addrs=127.0.97.62:41309
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:36:41.059736 9384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:36:41.061017 9384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:36:41.074321 9389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:41.074285 9390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:41.074854 9392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:36:41.075064 9384 server_base.cc:1034] running on GCE node
I20260620 23:36:41.075665 9384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:36:41.076597 9384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:36:41.077904 9384 hybrid_clock.cc:648] HybridClock initialized: now 1781998601077863 us; error 32 us; skew 500 ppm
I20260620 23:36:41.078224 9384 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:36:41 dist-test-slave-tcgx krb5kdc[8646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998601, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:36:41.081389 9384 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260620 23:36:41.082772 9384 webserver.cc:460] Webserver started at http://127.0.97.2:44045/ using document root <none> and password file <none>
I20260620 23:36:41.083549 9384 fs_manager.cc:359] Metadata directory not provided
I20260620 23:36:41.083667 9384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:36:41.089294 9384 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260620 23:36:41.092118 9399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:41.093532 9384 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 23:36:41.093709 9384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-1/wal
uuid: "7dd27215fbe4481bbf8235001d0b8c79"
format_stamp: "Formatted at 2026-06-20 23:36:38 on dist-test-slave-tcgx"
server_key: "3c0658d6165b0e97e53b9be947277354"
server_key_iv: "f72a3a08ccff2300ad16343901b61ebf"
server_key_version: "encryptionkey@0"
I20260620 23:36:41.094678 9384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:36:41.133148 9384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:36:41.138036 9384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:36:41.138340 9384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:36:41.139381 9384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:36:41.140887 9384 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:36:41.140985 9384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:41.141152 9384 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:36:41.141234 9384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:41.174896 9384 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40637
I20260620 23:36:41.174973 9511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40637 every 8 connection(s)
I20260620 23:36:41.176425 9384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
I20260620 23:36:41.181583 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 9384
I20260620 23:36:41.182303 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:36495
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:41309
--webserver_port=43239
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.3
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/krb5kdc/oryx_127.0.97.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:36:41 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:36:41 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 23:36:41.198734 9382 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:36:41.179430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52285 (local address 127.0.97.62:41309)
0620 23:36:41.179638 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:41.179644 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:41.181066 (+ 1422us) server_negotiation.cc:407] Connection header received
0620 23:36:41.183375 (+ 2309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:41.183382 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:41.183471 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:41.183543 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:41.185746 (+ 2203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:41.186853 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:41.191128 (+ 4275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:41.191783 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:41.191943 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:41.198539 (+ 6596us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:52285: BlockingRecv error: recv got EOF from 127.0.97.2:52285 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 23:36:41.199747 9514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:41.180044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41309 (local address 127.0.97.2:52285)
0620 23:36:41.180889 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:41.180935 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:41.183014 (+ 2079us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:41.184063 (+ 1049us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:41.184075 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:41.184789 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:41.185534 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:41.185557 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:41.187058 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:41.187063 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:41.190952 (+ 3889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:41.190973 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:41.192056 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:41.192066 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:41.192376 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:41.193195 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:41.193230 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:41.198204 (+ 4974us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41309: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":354,"threads_started":1}
W20260620 23:36:41.200565 9512 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:41309 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:41309: Client connection negotiation failed: client connection to 127.0.97.62:41309: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Jun 20 23:36:41 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:36:41 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:36:41.210335 9514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:41.201276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41309 (local address 127.0.97.2:48629)
0620 23:36:41.201480 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:41.201502 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:41.201622 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:41.202009 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:41.202014 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:41.202390 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:41.202548 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:41.202557 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:41.203763 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:41.203772 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:41.204786 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:41.204797 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:41.205525 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:41.205530 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:41.205719 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:41.206370 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:41.206390 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:41.209967 (+ 3577us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41309: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":62}
W20260620 23:36:41.210443 9382 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:36:41.201374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48629 (local address 127.0.97.62:41309)
0620 23:36:41.201542 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:41.201546 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:41.201566 (+ 20us) server_negotiation.cc:407] Connection header received
0620 23:36:41.201783 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:41.201787 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:41.201853 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:41.201981 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:41.202699 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:41.203607 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:41.204935 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:41.205362 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:41.205513 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:41.210194 (+ 4681us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:48629: BlockingRecv error: recv got EOF from 127.0.97.2:48629 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 20 23:36:41 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:36:41 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:36:41.220368 9514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:41.211212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41309 (local address 127.0.97.2:43677)
0620 23:36:41.211410 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:41.211431 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:41.211534 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:41.211815 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:41.211820 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:41.212227 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:41.212414 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:41.212425 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:41.213697 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:41.213703 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:41.215296 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:41.215313 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:41.216080 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:41.216086 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:41.216269 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:41.216856 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:41.216879 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:41.220038 (+ 3159us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41309: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":66}
W20260620 23:36:41.220466 9382 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:36:41.211404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43677 (local address 127.0.97.62:41309)
0620 23:36:41.211552 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:41.211556 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:41.211589 (+ 33us) server_negotiation.cc:407] Connection header received
0620 23:36:41.211636 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:41.211640 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:41.211697 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:41.211776 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:41.212579 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:41.213562 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:41.215470 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:41.215928 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:41.216042 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:41.220251 (+ 4209us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:43677: BlockingRecv error: recv got EOF from 127.0.97.2:43677 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260620 23:36:41.220676 9512 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 23:36:41.533233 9515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:36:41.533581 9515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:36:41.533752 9515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:36:41.541270 9515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:36:41.541438 9515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:36:41.541625 9515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 23:36:41.552098 9515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:36495
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=43239
--tserver_master_addrs=127.0.97.62:41309
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:36:41.553634 9515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:36:41.555410 9515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:36:41.569293 9520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:41.569324 9521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:41.569368 9523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:36:41.570150 9515 server_base.cc:1034] running on GCE node
I20260620 23:36:41.570798 9515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:36:41.571965 9515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:36:41.573177 9515 hybrid_clock.cc:648] HybridClock initialized: now 1781998601573131 us; error 40 us; skew 500 ppm
I20260620 23:36:41.573530 9515 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:36:41 dist-test-slave-tcgx krb5kdc[8646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998601, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:36:41.576853 9515 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260620 23:36:41.578481 9515 webserver.cc:460] Webserver started at http://127.0.97.3:43239/ using document root <none> and password file <none>
I20260620 23:36:41.579291 9515 fs_manager.cc:359] Metadata directory not provided
I20260620 23:36:41.579452 9515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:36:41.585479 9515 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 23:36:41.588634 9530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:41.590121 9515 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 23:36:41.590320 9515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-2/wal
uuid: "e57494d0e2c047f385d1880190def5cd"
format_stamp: "Formatted at 2026-06-20 23:36:38 on dist-test-slave-tcgx"
server_key: "3f01503f0075f5614a0d64c2a46b12c6"
server_key_iv: "d55838f169047c2f3b810d90bf6ac6bc"
server_key_version: "encryptionkey@0"
I20260620 23:36:41.591435 9515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:36:41.617450 9515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:36:41.621799 9515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:36:41.622066 9515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:36:41.623082 9515 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:36:41.624395 9515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:36:41.624500 9515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:41.624585 9515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:36:41.624640 9515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:41.658697 9515 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36495
I20260620 23:36:41.658764 9642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36495 every 8 connection(s)
I20260620 23:36:41.660235 9515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
I20260620 23:36:41.660809 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 9515
Jun 20 23:36:41 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:36:41 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:36:41.678274 9382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:41.661911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49502 (local address 127.0.97.62:41309)
0620 23:36:41.662104 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:41.662111 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:41.662618 (+ 507us) server_negotiation.cc:407] Connection header received
0620 23:36:41.662841 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:41.662847 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:41.662959 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:41.663066 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:41.664981 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:41.665710 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:41.667566 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:41.668021 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:41.668118 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:41.671521 (+ 3403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:41.671529 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:41.671545 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:41.671589 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:41.674885 (+ 3296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:41.675602 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:41.675609 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:41.675616 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:41.675779 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:36:41.676173 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:36:41.676177 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:36:41.676179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:36:41.676757 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:41.677092 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:41.677616 (+ 524us) server_negotiation.cc:299] Negotiation successful
0620 23:36:41.677893 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260620 23:36:41.679200 9646 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:36:41.663146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57447 (local address 127.0.97.62:41309)
0620 23:36:41.663631 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:41.663637 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:41.664818 (+ 1181us) server_negotiation.cc:407] Connection header received
0620 23:36:41.667212 (+ 2394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:41.667216 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:41.667294 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:41.667381 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:41.669393 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:41.670668 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:41.672414 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:41.673205 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:41.673374 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:41.678921 (+ 5547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:57447: BlockingRecv error: recv got EOF from 127.0.97.3:57447 (error 108)
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":249,"threads_started":1}
I20260620 23:36:41.680466 9647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:41.663751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41309 (local address 127.0.97.3:57447)
0620 23:36:41.664615 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:41.664667 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:41.666871 (+ 2204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:41.667662 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:41.667671 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:41.668288 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:41.669197 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:41.669219 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:41.670845 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:41.670849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:41.672251 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:41.672266 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:41.673503 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:41.673514 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:41.673788 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:41.674411 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:41.674432 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:41.679503 (+ 5071us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41309: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":304,"threads_started":1}
W20260620 23:36:41.681197 9643 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:41309 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:41309: Client connection negotiation failed: client connection to 127.0.97.62:41309: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Jun 20 23:36:41 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:36:41 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:36:41.690158 9647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:41.681783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41309 (local address 127.0.97.3:33111)
0620 23:36:41.681973 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:41.681993 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:41.682101 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:41.682438 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:41.682441 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:41.682764 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:41.682903 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:41.682910 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:41.684080 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:41.684086 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:41.685011 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:41.685023 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:41.685712 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:41.685718 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:41.685899 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:41.686579 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:41.686601 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:41.689957 (+ 3356us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41309: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":64}
W20260620 23:36:41.690462 9646 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:36:41.681951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33111 (local address 127.0.97.62:41309)
0620 23:36:41.682146 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:41.682151 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:41.682171 (+ 20us) server_negotiation.cc:407] Connection header received
0620 23:36:41.682223 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:41.682227 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:41.682306 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:41.682390 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:41.683090 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:41.683929 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:41.685155 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:41.685561 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:41.685698 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:41.690225 (+ 4527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:33111: BlockingRecv error: recv got EOF from 127.0.97.3:33111 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Jun 20 23:36:41 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:36:41 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:36:41.699538 9647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:41.691065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41309 (local address 127.0.97.3:56195)
0620 23:36:41.691233 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:41.691255 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:41.691368 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:41.691795 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:41.691836 (+ 41us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:41.692127 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:41.692284 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:41.692292 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:41.693549 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:41.693555 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:41.694472 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:41.694484 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:41.695375 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:41.695380 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:41.695571 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:41.696118 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:41.696135 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:41.699271 (+ 3136us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41309: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 23:36:41.699760 9646 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:36:41.691180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56195 (local address 127.0.97.62:41309)
0620 23:36:41.691457 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:41.691463 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:41.691494 (+ 31us) server_negotiation.cc:407] Connection header received
0620 23:36:41.691550 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:41.691555 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:41.691628 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:41.691737 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:41.692439 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:41.693388 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:41.694619 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:41.695196 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:41.695363 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:41.699498 (+ 4135us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:56195: BlockingRecv error: recv got EOF from 127.0.97.3:56195 (error 108)
Metrics: {"server-negotiator.queue_time_us":152}
W20260620 23:36:41.699852 9643 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 23:36:41 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:36:41 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:36:41.736719 9648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:41.727676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41309 (local address 127.0.97.1:59433)
0620 23:36:41.728131 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:41.728147 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:41.728269 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:41.728582 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:41.728585 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:41.729013 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:41.729198 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:41.729206 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:41.730432 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:41.730439 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:41.731493 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:41.731506 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:41.732248 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:41.732254 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:41.732435 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:41.733080 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:41.733098 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:41.736420 (+ 3322us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41309: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":247,"threads_started":1}
W20260620 23:36:41.736891 9646 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:36:41.727739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59433 (local address 127.0.97.62:41309)
0620 23:36:41.728007 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:41.728014 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:41.728261 (+ 247us) server_negotiation.cc:407] Connection header received
0620 23:36:41.728363 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:41.728367 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:41.728442 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:41.728532 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:41.729377 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:41.730285 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:41.731646 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:41.732064 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:41.732212 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:41.736655 (+ 4443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:59433: BlockingRecv error: recv got EOF from 127.0.97.1:59433 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
Jun 20 23:36:42 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:36:42 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:36:42.230005 9649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:42.221318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41309 (local address 127.0.97.2:49151)
0620 23:36:42.221887 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:42.221905 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:42.222021 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:42.222318 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:42.222321 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:42.222816 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:42.223103 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:42.223113 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:42.224093 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:42.224097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:42.225121 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:42.225133 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:42.225935 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:42.225942 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:42.226112 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:42.226788 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:42.226843 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:42.229781 (+ 2938us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41309: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":284,"threads_started":1}
W20260620 23:36:42.230219 9646 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:36:42.221372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49151 (local address 127.0.97.62:41309)
0620 23:36:42.221552 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:42.221557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:42.221985 (+ 428us) server_negotiation.cc:407] Connection header received
0620 23:36:42.222123 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:42.222127 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:42.222189 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:42.222277 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:42.223229 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:42.223946 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:42.225245 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:42.225679 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:42.225794 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:42.230001 (+ 4207us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:49151: BlockingRecv error: recv got EOF from 127.0.97.2:49151 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 20 23:36:42 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:36:42 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:36:42.715039 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:42.700568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41309 (local address 127.0.97.3:40979)
0620 23:36:42.701126 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:42.701146 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:42.701299 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:42.701756 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:42.701764 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:42.702489 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:42.702781 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:42.702796 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:42.703878 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:42.703885 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:42.705807 (+ 1922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:42.705827 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:42.706644 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:42.706653 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:42.706884 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:42.710505 (+ 3621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:42.710541 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:42.714583 (+ 4042us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41309: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":295,"threads_started":1}
W20260620 23:36:42.715155 9646 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:36:42.700609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40979 (local address 127.0.97.62:41309)
0620 23:36:42.700836 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:42.700844 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:42.701274 (+ 430us) server_negotiation.cc:407] Connection header received
0620 23:36:42.701518 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:42.701523 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:42.701589 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:42.701708 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:42.702927 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:42.703736 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:42.705983 (+ 2247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:42.706429 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:42.706544 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:42.714833 (+ 8289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:40979: BlockingRecv error: recv got EOF from 127.0.97.3:40979 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 20 23:36:42 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:36:42 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:36:42.747901 9651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:42.737852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41309 (local address 127.0.97.1:33305)
0620 23:36:42.738383 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:42.738400 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:42.738577 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:42.738988 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:42.738992 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:42.739483 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:42.739681 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:42.739701 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:42.740968 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:42.740977 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:42.742138 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:42.742152 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:42.743164 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:42.743170 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:42.743346 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:42.744052 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:42.744071 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:42.747533 (+ 3462us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41309: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":300,"threads_started":1}
W20260620 23:36:42.748049 9646 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:36:42.737878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33305 (local address 127.0.97.62:41309)
0620 23:36:42.738121 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:42.738129 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:42.738501 (+ 372us) server_negotiation.cc:407] Connection header received
0620 23:36:42.738740 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:42.738744 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:42.738808 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:42.738914 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:42.739835 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:42.740841 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:42.742283 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:42.742980 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:42.743148 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:42.747751 (+ 4603us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:33305: BlockingRecv error: recv got EOF from 127.0.97.1:33305 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 23:36:43 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:36:43 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:36:43.239655 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:43.231094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41309 (local address 127.0.97.2:35349)
0620 23:36:43.231641 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:43.231657 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:43.231777 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:43.232092 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:43.232095 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:43.232482 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:43.232622 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:43.232629 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:43.233709 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:43.233712 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:43.234772 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:43.234795 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:43.235721 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:43.235727 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:43.235891 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:43.236624 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:43.236643 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:43.239405 (+ 2762us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41309: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":314,"threads_started":1}
W20260620 23:36:43.239856 9646 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:36:43.231222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35349 (local address 127.0.97.62:41309)
0620 23:36:43.231419 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:43.231424 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:43.231733 (+ 309us) server_negotiation.cc:407] Connection header received
0620 23:36:43.231875 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:43.231879 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:43.231940 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:43.232070 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:43.232750 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:43.233538 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:43.234969 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:43.235565 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:43.235709 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:43.239610 (+ 3901us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:35349: BlockingRecv error: recv got EOF from 127.0.97.2:35349 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 20 23:36:43 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:36:43 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:36:43.724681 9653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:43.716068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41309 (local address 127.0.97.3:48329)
0620 23:36:43.716623 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:43.716640 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:43.716763 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:43.717067 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:43.717070 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:43.717468 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:43.717611 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:43.717619 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:43.718658 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:43.718663 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:43.719736 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:43.719752 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:43.720486 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:43.720490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:43.720599 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:43.721255 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:43.721278 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:43.724441 (+ 3163us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41309: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":328,"threads_started":1}
W20260620 23:36:43.724908 9646 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:36:43.716241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48329 (local address 127.0.97.62:41309)
0620 23:36:43.716419 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:43.716424 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:43.716711 (+ 287us) server_negotiation.cc:407] Connection header received
0620 23:36:43.716873 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:43.716877 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:43.716938 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:43.717024 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:43.717750 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:43.718514 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:43.719912 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:43.720324 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:43.720463 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:43.724636 (+ 4173us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:48329: BlockingRecv error: recv got EOF from 127.0.97.3:48329 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 20 23:36:43 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:36:43 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:36:43.757157 9654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:43.748964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41309 (local address 127.0.97.1:53105)
0620 23:36:43.749494 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:43.749511 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:43.749621 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:43.749935 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:43.749938 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:43.750273 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:43.750472 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:43.750482 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:43.751650 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:43.751654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:43.752581 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:43.752593 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:43.753220 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:43.753225 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:43.753359 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:43.753941 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:43.753959 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:43.756774 (+ 2815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41309: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":292,"threads_started":1}
W20260620 23:36:43.757241 9646 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:36:43.749057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53105 (local address 127.0.97.62:41309)
0620 23:36:43.749234 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:43.749238 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:43.749598 (+ 360us) server_negotiation.cc:407] Connection header received
0620 23:36:43.749720 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:43.749724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:43.749802 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:43.749894 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:43.750633 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:43.751466 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:43.752713 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:43.753090 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:43.753181 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:43.757004 (+ 3823us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:53105: BlockingRecv error: recv got EOF from 127.0.97.1:53105 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 23:36:44 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:36:44 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:36:44.249280 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:44.240731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41309 (local address 127.0.97.2:47827)
0620 23:36:44.241231 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:44.241247 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:44.241431 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:44.241784 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:44.241788 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:44.242153 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:44.242322 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:44.242330 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:44.243415 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:44.243419 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:44.244657 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:44.244673 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:44.245434 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:44.245440 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:44.245625 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:44.246218 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:44.246236 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:44.249009 (+ 2773us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41309: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":271,"threads_started":1}
W20260620 23:36:44.249472 9646 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:36:44.240817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47827 (local address 127.0.97.62:41309)
0620 23:36:44.241089 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:44.241097 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:44.241351 (+ 254us) server_negotiation.cc:407] Connection header received
0620 23:36:44.241582 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:44.241586 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:44.241648 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:44.241755 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:44.242516 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:44.243257 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:44.244828 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:44.245282 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:44.245422 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:44.249215 (+ 3793us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:47827: BlockingRecv error: recv got EOF from 127.0.97.2:47827 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
Jun 20 23:36:44 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:36:44 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:36:44.734223 9656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:44.725763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41309 (local address 127.0.97.3:50597)
0620 23:36:44.726370 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:44.726386 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:44.726553 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:44.726923 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:44.726928 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:44.727314 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:44.727483 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:44.727492 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:44.728539 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:44.728544 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:44.729472 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:44.729484 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:44.730192 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:44.730198 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:44.730381 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:44.731053 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:44.731070 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:44.733966 (+ 2896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41309: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":360,"threads_started":1}
W20260620 23:36:44.734397 9646 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:36:44.725852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50597 (local address 127.0.97.62:41309)
0620 23:36:44.726085 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:44.726092 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:44.726481 (+ 389us) server_negotiation.cc:407] Connection header received
0620 23:36:44.726690 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:44.726695 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:44.726773 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:44.726877 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:44.727636 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:44.728377 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:44.729619 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:44.730037 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:44.730168 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:44.734181 (+ 4013us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:50597: BlockingRecv error: recv got EOF from 127.0.97.3:50597 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Jun 20 23:36:44 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:36:44 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:36:44.766892 9657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:44.758232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41309 (local address 127.0.97.1:60699)
0620 23:36:44.758733 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:44.758748 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:44.758859 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:44.759305 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:44.759308 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:44.759665 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:44.759816 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:44.759824 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:44.760802 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:44.760806 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:44.761998 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:44.762011 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:44.762740 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:44.762745 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:44.763005 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:44.763668 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:44.763685 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:44.766548 (+ 2863us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41309: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":279,"threads_started":1}
W20260620 23:36:44.767161 9646 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:36:44.758303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60699 (local address 127.0.97.62:41309)
0620 23:36:44.758527 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:44.758532 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:44.758821 (+ 289us) server_negotiation.cc:407] Connection header received
0620 23:36:44.759109 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:44.759113 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:44.759176 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:44.759261 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:44.759942 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:44.760661 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:44.762155 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:44.762587 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:44.762729 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:44.766751 (+ 4022us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:60699: BlockingRecv error: recv got EOF from 127.0.97.1:60699 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Jun 20 23:36:45 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:36:45 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:36:45.258272 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:45.250295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41309 (local address 127.0.97.2:50093)
0620 23:36:45.250758 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:45.250774 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:45.250883 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:45.251213 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:45.251217 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:45.251573 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:45.251715 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:45.251723 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:45.252712 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:45.252715 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:45.253666 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:45.253677 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:45.254367 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:45.254371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:45.254505 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:45.255192 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:45.255210 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:45.258039 (+ 2829us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41309: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":249,"threads_started":1}
W20260620 23:36:45.258540 9646 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:36:45.250357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50093 (local address 127.0.97.62:41309)
0620 23:36:45.250560 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:45.250570 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:45.250851 (+ 281us) server_negotiation.cc:407] Connection header received
0620 23:36:45.251027 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:45.251031 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:45.251094 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:45.251171 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:45.251838 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:45.252606 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:45.253808 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:45.254228 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:45.254327 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:45.258279 (+ 3952us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:50093: BlockingRecv error: recv got EOF from 127.0.97.2:50093 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 23:36:45 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:36:45 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:36:45.743975 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:45.735307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41309 (local address 127.0.97.3:59743)
0620 23:36:45.735844 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:45.735861 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:45.736004 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:45.736383 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:45.736387 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:45.736922 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:45.737135 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:45.737145 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:45.738132 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:45.738139 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:45.739191 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:45.739203 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:45.739894 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:45.739900 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:45.740082 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:45.740686 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:45.740703 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:45.743754 (+ 3051us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41309: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":335,"threads_started":1}
W20260620 23:36:45.744192 9646 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:36:45.735482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59743 (local address 127.0.97.62:41309)
0620 23:36:45.735646 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:45.735652 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:45.735949 (+ 297us) server_negotiation.cc:407] Connection header received
0620 23:36:45.736142 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:45.736146 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:45.736237 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:45.736350 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:45.737276 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:45.738001 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:45.739332 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:45.739738 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:45.739871 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:45.743952 (+ 4081us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:59743: BlockingRecv error: recv got EOF from 127.0.97.3:59743 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Jun 20 23:36:45 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:36:45 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:36:45.776499 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:45.767890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41309 (local address 127.0.97.1:45829)
0620 23:36:45.768443 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:45.768459 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:45.768631 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:45.768975 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:45.768980 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:45.769403 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:45.769577 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:45.769586 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:45.770565 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:45.770572 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:45.771828 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:45.771845 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:45.772572 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:45.772577 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:45.772751 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:45.773376 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:45.773393 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:45.776253 (+ 2860us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41309: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":280,"threads_started":1}
W20260620 23:36:45.776682 9646 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:36:45.768056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45829 (local address 127.0.97.62:41309)
0620 23:36:45.768296 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:45.768303 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:45.768559 (+ 256us) server_negotiation.cc:407] Connection header received
0620 23:36:45.768776 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:45.768781 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:45.768843 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:45.768944 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:45.769720 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:45.770428 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:45.771951 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:45.772423 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:45.772559 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:45.776455 (+ 3896us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:45829: BlockingRecv error: recv got EOF from 127.0.97.1:45829 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Jun 20 23:36:46 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:36:46 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:36:46.267633 9661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:46.259314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41309 (local address 127.0.97.2:33207)
0620 23:36:46.259799 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:46.259816 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:46.259938 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:46.260242 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:46.260245 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:46.260645 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:46.260808 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:46.260816 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:46.261849 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:46.261853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:46.262779 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:46.262791 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:46.263488 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:46.263492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:46.263610 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:46.264200 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:46.264218 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:46.267325 (+ 3107us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41309: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":274,"threads_started":1}
W20260620 23:36:46.267795 9646 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:36:46.259548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33207 (local address 127.0.97.62:41309)
0620 23:36:46.259752 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:46.259759 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:46.259892 (+ 133us) server_negotiation.cc:407] Connection header received
0620 23:36:46.260052 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:46.260055 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:46.260116 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:46.260196 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:46.260960 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:46.261703 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:46.262914 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:46.263348 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:46.263451 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:46.267535 (+ 4084us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:33207: BlockingRecv error: recv got EOF from 127.0.97.2:33207 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Jun 20 23:36:46 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:36:46 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:36:46.753549 9662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:46.744985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41309 (local address 127.0.97.3:45349)
0620 23:36:46.745523 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:46.745540 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:46.745652 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:46.745959 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:46.745962 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:46.746336 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:46.746495 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:46.746503 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:46.747550 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:46.747553 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:46.748764 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:46.748780 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:46.749605 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:46.749611 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:46.749760 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:46.750396 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:46.750413 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:46.753288 (+ 2875us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41309: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":267,"threads_started":1}
W20260620 23:36:46.753765 9646 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:36:46.745131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45349 (local address 127.0.97.62:41309)
0620 23:36:46.745309 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:46.745314 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:46.745618 (+ 304us) server_negotiation.cc:407] Connection header received
0620 23:36:46.745756 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:46.745760 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:46.745822 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:46.745916 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:46.746655 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:46.747414 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:46.748914 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:46.749458 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:46.749580 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:46.753514 (+ 3934us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:45349: BlockingRecv error: recv got EOF from 127.0.97.3:45349 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 20 23:36:46 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:36:46 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:36:46.786021 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:46.777431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41309 (local address 127.0.97.1:54953)
0620 23:36:46.778015 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:46.778031 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:46.778146 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:46.778432 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:46.778436 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:46.778798 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:46.778993 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:46.779001 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:46.779980 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:46.779984 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:46.780900 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:46.780910 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:46.781651 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:46.781655 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:46.781774 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:46.782357 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:46.782373 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:46.785661 (+ 3288us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41309: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":379,"threads_started":1}
W20260620 23:36:46.786172 9646 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:36:46.777495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54953 (local address 127.0.97.62:41309)
0620 23:36:46.777663 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:46.777667 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:46.778114 (+ 447us) server_negotiation.cc:407] Connection header received
0620 23:36:46.778241 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:46.778244 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:46.778307 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:46.778390 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:46.779124 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:46.779843 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:46.781045 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:46.781515 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:46.781619 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:46.785903 (+ 4284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:54953: BlockingRecv error: recv got EOF from 127.0.97.1:54953 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 20 23:36:47 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:36:47 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:36:47.276417 9664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:47.268720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41309 (local address 127.0.97.2:45799)
0620 23:36:47.269231 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:47.269248 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:47.269355 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:47.269641 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:47.269644 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:47.270001 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:47.270148 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:47.270156 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:47.271107 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:47.271110 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:47.272022 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:47.272035 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:47.272831 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:47.272835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:47.272946 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:47.273507 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:47.273529 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:47.276211 (+ 2682us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41309: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":298,"threads_started":1}
W20260620 23:36:47.276580 9646 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:36:47.268772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45799 (local address 127.0.97.62:41309)
0620 23:36:47.268970 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:47.268974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:47.269324 (+ 350us) server_negotiation.cc:407] Connection header received
0620 23:36:47.269454 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:47.269457 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:47.269520 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:47.269602 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:47.270271 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:47.270989 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:47.272145 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:47.272690 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:47.272797 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:47.276404 (+ 3607us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:45799: BlockingRecv error: recv got EOF from 127.0.97.2:45799 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 20 23:36:47 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:36:47 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:36:47.768822 9665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:47.754493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41309 (local address 127.0.97.3:41865)
0620 23:36:47.755117 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:47.755133 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:47.755305 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:47.755669 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:47.755673 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:47.756077 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:47.756238 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:47.756246 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:47.757354 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:47.757360 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:47.758270 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:47.758281 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:47.759296 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:47.759301 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:47.759428 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:47.765307 (+ 5879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:47.765329 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:47.768601 (+ 3272us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41309: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":322,"threads_started":1}
W20260620 23:36:47.769156 9646 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:36:47.754686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41865 (local address 127.0.97.62:41309)
0620 23:36:47.754957 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:47.754966 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:47.755230 (+ 264us) server_negotiation.cc:407] Connection header received
0620 23:36:47.755446 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:47.755452 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:47.755526 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:47.755637 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:47.756413 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:47.757210 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:47.758418 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:47.758851 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:47.759034 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:47.768855 (+ 9821us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:41865: BlockingRecv error: recv got EOF from 127.0.97.3:41865 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Jun 20 23:36:47 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:36:47 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:36:47.795552 9666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:47.787006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41309 (local address 127.0.97.1:38199)
0620 23:36:47.787599 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:47.787615 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:47.787781 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:47.788124 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:47.788128 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:47.788560 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:47.788721 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:47.788729 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:47.789726 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:47.789732 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:47.790661 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:47.790671 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:47.791630 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:47.791635 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:47.791811 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:47.792339 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:47.792357 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:47.795174 (+ 2817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41309: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":275,"threads_started":1}
W20260620 23:36:47.795657 9646 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:36:47.787153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38199 (local address 127.0.97.62:41309)
0620 23:36:47.787349 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:47.787358 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:47.787708 (+ 350us) server_negotiation.cc:407] Connection header received
0620 23:36:47.787906 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:47.787912 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:47.787986 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:47.788096 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:47.788874 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:47.789580 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:47.790814 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:47.791472 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:47.791622 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:47.795426 (+ 3804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:38199: BlockingRecv error: recv got EOF from 127.0.97.1:38199 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 20 23:36:48 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:36:48 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:36:48.286358 9667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:48.277665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41309 (local address 127.0.97.2:35063)
0620 23:36:48.278440 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:48.278460 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:48.278601 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:48.278967 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:48.278972 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:48.279432 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:48.279641 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:48.279650 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:48.280622 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:48.280631 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:48.281541 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:48.281553 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:48.282316 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:48.282322 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:48.282489 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:48.283181 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:48.283199 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:48.286126 (+ 2927us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41309: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":407,"threads_started":1}
W20260620 23:36:48.286588 9646 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:36:48.277766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35063 (local address 127.0.97.62:41309)
0620 23:36:48.278022 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:48.278030 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:48.278566 (+ 536us) server_negotiation.cc:407] Connection header received
0620 23:36:48.278733 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:48.278744 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:48.278803 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:48.278906 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:48.279780 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:48.280470 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:48.281687 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:48.282162 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:48.282303 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:48.286353 (+ 4050us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:35063: BlockingRecv error: recv got EOF from 127.0.97.2:35063 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
Jun 20 23:36:48 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:36:48 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:36:48.778280 9668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:48.769815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41309 (local address 127.0.97.3:34131)
0620 23:36:48.770388 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:48.770404 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:48.770521 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:48.770871 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:48.770875 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:48.771309 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:48.771488 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:48.771497 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:48.772627 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:48.772633 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:48.773586 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:48.773597 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:48.774294 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:48.774300 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:48.774460 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:48.775122 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:48.775140 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:48.777925 (+ 2785us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41309: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":277,"threads_started":1}
W20260620 23:36:48.778337 9646 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:36:48.769993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34131 (local address 127.0.97.62:41309)
0620 23:36:48.770258 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:48.770264 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:48.770485 (+ 221us) server_negotiation.cc:407] Connection header received
0620 23:36:48.770630 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:48.770634 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:48.770694 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:48.770777 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:48.771658 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:48.772472 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:48.773730 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:48.774145 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:48.774280 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:48.778113 (+ 3833us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:34131: BlockingRecv error: recv got EOF from 127.0.97.3:34131 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Jun 20 23:36:48 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:36:48 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:36:48.805938 9669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:48.796604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41309 (local address 127.0.97.1:43827)
0620 23:36:48.797133 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:48.797149 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:48.797303 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:48.797652 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:48.797658 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:48.798144 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:48.798348 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:48.798357 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:48.799509 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:48.799513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:48.800444 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:48.800455 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:48.801390 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:48.801396 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:48.801619 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:48.802397 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:48.802428 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:48.805710 (+ 3282us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41309: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":279,"threads_started":1}
W20260620 23:36:48.806242 9646 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:36:48.796745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43827 (local address 127.0.97.62:41309)
0620 23:36:48.796894 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:48.796898 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:48.797235 (+ 337us) server_negotiation.cc:407] Connection header received
0620 23:36:48.797429 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:48.797435 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:48.797510 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:48.797592 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:48.798505 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:48.799388 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:48.800570 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:48.801208 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:48.801344 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:48.805950 (+ 4606us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:43827: BlockingRecv error: recv got EOF from 127.0.97.1:43827 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 20 23:36:49 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:36:49 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:36:49.296499 9670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:49.287364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41309 (local address 127.0.97.2:36193)
0620 23:36:49.287929 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:49.287946 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:49.288124 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:49.288490 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:49.288495 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:49.288929 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:49.289083 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:49.289092 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:49.290269 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:49.290275 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:49.291460 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:49.291472 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:49.292418 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:49.292424 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:49.292586 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:49.293247 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:49.293267 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:49.296269 (+ 3002us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41309: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":298,"threads_started":1}
W20260620 23:36:49.296909 9646 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:36:49.287549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36193 (local address 127.0.97.62:41309)
0620 23:36:49.287799 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:49.287805 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:49.288048 (+ 243us) server_negotiation.cc:407] Connection header received
0620 23:36:49.288276 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:49.288281 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:49.288343 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:49.288457 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:49.289247 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:49.290125 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:49.291610 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:49.292229 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:49.292366 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:49.296629 (+ 4263us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:36193: BlockingRecv error: recv got EOF from 127.0.97.2:36193 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Jun 20 23:36:49 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:36:49 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:36:49.787531 9671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:49.779236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41309 (local address 127.0.97.3:56449)
0620 23:36:49.779758 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:49.779772 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:49.779927 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:49.780276 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:49.780280 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:49.780681 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:49.780830 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:49.780838 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:49.781803 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:49.781809 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:49.782708 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:49.782719 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:49.783499 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:49.783505 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:49.783680 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:49.784362 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:49.784379 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:49.787197 (+ 2818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41309: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":283,"threads_started":1}
W20260620 23:36:49.787637 9646 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:36:49.779315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56449 (local address 127.0.97.62:41309)
0620 23:36:49.779531 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:49.779539 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:49.779863 (+ 324us) server_negotiation.cc:407] Connection header received
0620 23:36:49.780058 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:49.780082 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:49.780141 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:49.780245 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:49.780972 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:49.781664 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:49.782845 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:49.783321 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:49.783463 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:49.787414 (+ 3951us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:56449: BlockingRecv error: recv got EOF from 127.0.97.3:56449 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 20 23:36:49 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:36:49 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:36:49.816047 9672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:49.807034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41309 (local address 127.0.97.1:45605)
0620 23:36:49.807776 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:49.807797 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:49.807932 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:49.808262 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:49.808267 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:49.808780 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:49.809026 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:49.809037 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:49.810358 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:49.810362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:49.811416 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:49.811428 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:49.812083 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:49.812087 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:49.812204 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:49.812750 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:49.812767 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:49.815697 (+ 2930us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41309: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":349,"threads_started":1}
W20260620 23:36:49.816129 9646 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:36:49.807573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45605 (local address 127.0.97.62:41309)
0620 23:36:49.807785 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:49.807794 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:49.807869 (+ 75us) server_negotiation.cc:407] Connection header received
0620 23:36:49.808042 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:49.808047 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:49.808110 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:49.808188 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:49.809214 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:49.810229 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:49.811550 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:49.811959 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:49.812053 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:49.815885 (+ 3832us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:45605: BlockingRecv error: recv got EOF from 127.0.97.1:45605 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 23:36:50 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:36:50 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:36:50.306433 9673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:50.297484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41309 (local address 127.0.97.2:37417)
0620 23:36:50.298071 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:50.298087 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:50.298235 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:50.298544 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:50.298547 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:50.299021 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:50.299290 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:50.299303 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:50.300434 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:50.300439 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:50.301762 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:50.301774 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:50.302436 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:50.302441 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:50.302561 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:50.303212 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:50.303229 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:50.306187 (+ 2958us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41309: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":336,"threads_started":1}
W20260620 23:36:50.306627 9646 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:36:50.297531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37417 (local address 127.0.97.62:41309)
0620 23:36:50.297695 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:50.297700 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:50.298197 (+ 497us) server_negotiation.cc:407] Connection header received
0620 23:36:50.298354 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:50.298358 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:50.298420 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:50.298504 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:50.299432 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:50.300289 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:50.301888 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:50.302293 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:50.302388 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:50.306371 (+ 3983us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:37417: BlockingRecv error: recv got EOF from 127.0.97.2:37417 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 23:36:50 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:36:50 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:36:50.797871 9674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:50.788658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41309 (local address 127.0.97.3:40433)
0620 23:36:50.789217 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:50.789235 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:50.789362 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:50.789759 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:50.789763 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:50.790315 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:50.790542 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:50.790558 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:50.791764 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:50.791769 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:50.792748 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:50.792760 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:50.793475 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:50.793479 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:50.793603 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:50.794213 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:50.794233 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:50.797544 (+ 3311us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41309: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":316,"threads_started":1}
W20260620 23:36:50.798076 9646 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:36:50.788727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40433 (local address 127.0.97.62:41309)
0620 23:36:50.789017 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:50.789024 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:50.789334 (+ 310us) server_negotiation.cc:407] Connection header received
0620 23:36:50.789495 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:50.789502 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:50.789580 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:50.789716 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:50.790708 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:50.791564 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:50.792897 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:50.793317 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:50.793422 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:50.797840 (+ 4418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:40433: BlockingRecv error: recv got EOF from 127.0.97.3:40433 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Jun 20 23:36:50 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:36:50 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:36:50.825963 9675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:50.817040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41309 (local address 127.0.97.1:44319)
0620 23:36:50.817656 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:50.817673 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:50.817786 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:50.818134 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:50.818138 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:50.818531 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:50.818699 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:50.818707 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:50.819847 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:50.819850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:50.820787 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:50.820799 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:50.821564 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:50.821569 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:50.821712 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:50.822415 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:50.822433 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:50.825628 (+ 3195us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41309: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":383,"threads_started":1}
W20260620 23:36:50.826297 9646 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:36:50.817164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44319 (local address 127.0.97.62:41309)
0620 23:36:50.817417 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:50.817422 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:50.817756 (+ 334us) server_negotiation.cc:407] Connection header received
0620 23:36:50.817892 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:50.817896 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:50.817958 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:50.818056 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:50.818848 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:50.819714 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:50.820926 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:50.821410 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:50.821513 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:50.825858 (+ 4345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:44319: BlockingRecv error: recv got EOF from 127.0.97.1:44319 (error 108)
Metrics: {"server-negotiator.queue_time_us":116}
Jun 20 23:36:51 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:36:51 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:36:51.316458 9676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:51.307604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41309 (local address 127.0.97.2:35833)
0620 23:36:51.308152 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:51.308169 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:51.308284 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:51.308598 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:51.308601 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:51.309134 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:51.309315 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:51.309324 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:51.310707 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:51.310713 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:51.311926 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:51.311938 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:51.312634 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:51.312638 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:51.312758 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:51.313351 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:51.313369 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:51.316215 (+ 2846us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41309: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":328,"threads_started":1}
W20260620 23:36:51.316674 9646 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:36:51.307700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35833 (local address 127.0.97.62:41309)
0620 23:36:51.307898 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:51.307903 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:51.308258 (+ 355us) server_negotiation.cc:407] Connection header received
0620 23:36:51.308392 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:51.308396 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:51.308461 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:51.308554 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:51.309471 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:51.310515 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:51.312059 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:51.312476 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:51.312581 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:51.316411 (+ 3830us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:35833: BlockingRecv error: recv got EOF from 127.0.97.2:35833 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Jun 20 23:36:51 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:36:51 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:36:51.807967 9677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:51.798881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41309 (local address 127.0.97.3:55347)
0620 23:36:51.799471 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:51.799488 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:51.799644 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:51.800063 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:51.800068 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:51.800467 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:51.800605 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:51.800613 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:51.801662 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:51.801669 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:51.802747 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:51.802782 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:51.803675 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:51.803682 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:51.803840 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:51.804573 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:51.804598 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:51.807633 (+ 3035us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41309: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":377,"threads_started":1}
W20260620 23:36:51.808130 9646 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:36:51.799083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55347 (local address 127.0.97.62:41309)
0620 23:36:51.799361 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:51.799368 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:51.799574 (+ 206us) server_negotiation.cc:407] Connection header received
0620 23:36:51.799811 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:51.799817 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:51.799888 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:51.799997 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:51.800766 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:51.801527 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:51.802913 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:51.803522 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:51.803681 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:51.807839 (+ 4158us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:55347: BlockingRecv error: recv got EOF from 127.0.97.3:55347 (error 108)
Metrics: {"server-negotiator.queue_time_us":132}
Jun 20 23:36:51 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:36:51 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:36:51.835898 9678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:51.827129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41309 (local address 127.0.97.1:38321)
0620 23:36:51.827656 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:51.827677 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:51.827822 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:51.828218 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:51.828222 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:51.828782 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:51.829018 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:51.829030 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:51.830028 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:51.830032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:51.831021 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:51.831037 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:51.831809 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:51.831813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:51.831933 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:51.832549 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:51.832568 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:51.835626 (+ 3058us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41309: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":272,"threads_started":1}
W20260620 23:36:51.836045 9646 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:36:51.827210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38321 (local address 127.0.97.62:41309)
0620 23:36:51.827457 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:51.827463 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:51.827769 (+ 306us) server_negotiation.cc:407] Connection header received
0620 23:36:51.827953 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:51.827958 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:51.828042 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:51.828150 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:51.829163 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:51.829908 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:51.831177 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:51.831662 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:51.831767 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:51.835823 (+ 4056us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:38321: BlockingRecv error: recv got EOF from 127.0.97.1:38321 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Jun 20 23:36:52 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:36:52 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:36:52.326537 9679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:52.317680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41309 (local address 127.0.97.2:51605)
0620 23:36:52.318220 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:52.318238 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:52.318359 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:52.318694 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:52.318697 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:52.319254 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:52.319492 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:52.319501 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:52.320677 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:52.320680 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:52.321565 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:52.321575 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:52.322434 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:52.322440 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:52.322615 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:52.323331 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:52.323349 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:52.326164 (+ 2815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41309: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":290,"threads_started":1}
W20260620 23:36:52.326656 9646 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:36:52.317830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51605 (local address 127.0.97.62:41309)
0620 23:36:52.318123 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:52.318134 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:52.318326 (+ 192us) server_negotiation.cc:407] Connection header received
0620 23:36:52.318476 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:52.318480 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:52.318545 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:52.318652 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:52.319647 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:52.320537 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:52.321725 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:52.322273 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:52.322405 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:52.326405 (+ 4000us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:51605: BlockingRecv error: recv got EOF from 127.0.97.2:51605 (error 108)
Metrics: {"server-negotiator.queue_time_us":159}
Jun 20 23:36:52 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:36:52 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:36:52.824501 9680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:52.808999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41309 (local address 127.0.97.3:59219)
0620 23:36:52.809594 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:52.809610 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:52.809774 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:52.810172 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:52.810178 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:52.810614 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:52.810787 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:52.810795 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:52.811816 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:52.811824 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:52.812723 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:52.812734 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:52.813446 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:52.813451 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:52.813625 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:52.821180 (+ 7555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:52.821203 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:52.824135 (+ 2932us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41309: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":313,"threads_started":1}
W20260620 23:36:52.824626 9646 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:36:52.809187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59219 (local address 127.0.97.62:41309)
0620 23:36:52.809448 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:52.809453 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:52.809700 (+ 247us) server_negotiation.cc:407] Connection header received
0620 23:36:52.809912 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:52.809917 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:52.810026 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:52.810135 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:52.810958 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:52.811687 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:52.812862 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:52.813287 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:52.813415 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:52.824387 (+ 10972us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:59219: BlockingRecv error: recv got EOF from 127.0.97.3:59219 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Jun 20 23:36:52 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:36:52 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:36:52.845613 9681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:52.836869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41309 (local address 127.0.97.1:43061)
0620 23:36:52.837467 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:52.837484 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:52.837601 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:52.838003 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:52.838008 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:52.838386 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:52.838549 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:52.838557 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:52.839611 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:52.839615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:52.840580 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:52.840592 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:52.841338 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:52.841344 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:52.841556 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:52.842212 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:52.842232 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:52.845382 (+ 3150us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41309: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":348,"threads_started":1}
W20260620 23:36:52.845834 9646 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:36:52.836949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43061 (local address 127.0.97.62:41309)
0620 23:36:52.837175 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:52.837179 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:52.837571 (+ 392us) server_negotiation.cc:407] Connection header received
0620 23:36:52.837720 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:52.837725 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:52.837807 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:52.837929 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:52.838680 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:52.839474 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:52.840712 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:52.841171 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:52.841264 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:52.845566 (+ 4302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:43061: BlockingRecv error: recv got EOF from 127.0.97.1:43061 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Jun 20 23:36:53 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:36:53 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:36:53.336408 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:53.327613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41309 (local address 127.0.97.2:37647)
0620 23:36:53.328204 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:53.328220 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:53.328331 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:53.328656 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:53.328659 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:53.329068 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:53.329219 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:53.329227 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:53.330230 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:53.330234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:53.331311 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:53.331323 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:53.332114 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:53.332119 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:53.332264 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:53.332929 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:53.332948 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:53.336159 (+ 3211us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41309: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":294,"threads_started":1}
W20260620 23:36:53.336608 9646 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:36:53.327601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37647 (local address 127.0.97.62:41309)
0620 23:36:53.327799 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:53.327807 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:53.328305 (+ 498us) server_negotiation.cc:407] Connection header received
0620 23:36:53.328445 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:53.328449 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:53.328513 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:53.328596 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:53.329349 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:53.330076 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:53.331476 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:53.331949 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:53.332072 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:53.336356 (+ 4284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:37647: BlockingRecv error: recv got EOF from 127.0.97.2:37647 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 20 23:36:53 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:36:53 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:36:53.834021 9683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:53.825594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41309 (local address 127.0.97.3:58133)
0620 23:36:53.826162 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:53.826180 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:53.826297 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:53.826779 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:53.826784 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:53.827265 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:53.827425 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:53.827434 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:53.828457 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:53.828464 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:53.829382 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:53.829393 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:53.830105 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:53.830110 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:53.830292 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:53.830919 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:53.831002 (+ 83us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:53.833786 (+ 2784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41309: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":299,"threads_started":1}
W20260620 23:36:53.834317 9646 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:36:53.825645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58133 (local address 127.0.97.62:41309)
0620 23:36:53.825807 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:53.825811 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:53.826251 (+ 440us) server_negotiation.cc:407] Connection header received
0620 23:36:53.826517 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:53.826523 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:53.826631 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:53.826736 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:53.827569 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:53.828314 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:53.829527 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:53.829952 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:53.830077 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:53.834016 (+ 3939us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:58133: BlockingRecv error: recv got EOF from 127.0.97.3:58133 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 23:36:53 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:36:53 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:36:53.855278 9684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:53.846714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41309 (local address 127.0.97.1:49459)
0620 23:36:53.847384 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:53.847400 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:53.847541 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:53.847919 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:53.847923 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:53.848380 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:53.848543 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:53.848561 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:53.849539 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:53.849543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:53.850460 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:53.850477 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:53.851363 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:53.851368 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:53.851485 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:53.852146 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:53.852165 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:53.855033 (+ 2868us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41309: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":280,"threads_started":1}
W20260620 23:36:53.855619 9646 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:36:53.846787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49459 (local address 127.0.97.62:41309)
0620 23:36:53.847077 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:53.847083 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:53.847497 (+ 414us) server_negotiation.cc:407] Connection header received
0620 23:36:53.847683 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:53.847689 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:53.847769 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:53.847874 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:53.848685 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:53.849415 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:53.850643 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:53.851190 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:53.851321 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:53.855240 (+ 3919us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:49459: BlockingRecv error: recv got EOF from 127.0.97.1:49459 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Jun 20 23:36:54 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:36:54 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:36:54.346264 9685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:54.337455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41309 (local address 127.0.97.2:51691)
0620 23:36:54.337969 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:54.337986 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:54.338100 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:54.338397 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:54.338400 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:54.338808 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:54.338988 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:54.338996 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:54.340134 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:54.340139 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:54.341071 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:54.341083 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:54.341837 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:54.341841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:54.341955 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:54.342657 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:54.342711 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:54.345984 (+ 3273us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41309: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":275,"threads_started":1}
W20260620 23:36:54.346467 9646 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:36:54.337648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51691 (local address 127.0.97.62:41309)
0620 23:36:54.337881 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:54.337888 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:54.338067 (+ 179us) server_negotiation.cc:407] Connection header received
0620 23:36:54.338207 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:54.338211 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:54.338274 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:54.338359 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:54.339151 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:54.339988 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:54.341216 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:54.341677 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:54.341781 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:54.346248 (+ 4467us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:51691: BlockingRecv error: recv got EOF from 127.0.97.2:51691 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Jun 20 23:36:54 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:36:54 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:36:54.844310 9686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:54.835318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41309 (local address 127.0.97.3:59203)
0620 23:36:54.835875 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:54.835912 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:54.836053 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:54.836416 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:54.836420 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:54.836924 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:54.837103 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:54.837112 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:54.838221 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:54.838228 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:54.839308 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:54.839320 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:54.840112 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:54.840120 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:54.840284 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:54.840963 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:54.840981 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:54.843971 (+ 2990us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41309: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":256,"threads_started":1}
W20260620 23:36:54.844373 9646 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:36:54.835398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59203 (local address 127.0.97.62:41309)
0620 23:36:54.835593 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:54.835601 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:54.836003 (+ 402us) server_negotiation.cc:407] Connection header received
0620 23:36:54.836208 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:54.836212 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:54.836272 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:54.836384 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:54.837257 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:54.838072 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:54.839468 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:54.839920 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:54.840075 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:54.844155 (+ 4080us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:59203: BlockingRecv error: recv got EOF from 127.0.97.3:59203 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 23:36:54 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:36:54 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:36:54.865064 9687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:54.856348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41309 (local address 127.0.97.1:42565)
0620 23:36:54.856900 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:54.856917 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:54.857123 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:54.857497 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:54.857503 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:54.857905 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:54.858080 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:54.858088 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:54.859142 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:54.859148 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:54.860243 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:54.860255 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:54.861167 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:54.861173 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:54.861345 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:54.861931 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:54.861950 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:54.864794 (+ 2844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41309: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":311,"threads_started":1}
W20260620 23:36:54.865389 9646 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:36:54.856456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42565 (local address 127.0.97.62:41309)
0620 23:36:54.856815 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:54.856822 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:54.857043 (+ 221us) server_negotiation.cc:407] Connection header received
0620 23:36:54.857263 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:54.857275 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:54.857351 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:54.857457 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:54.858223 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:54.858929 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:54.860386 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:54.861012 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:54.861142 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:54.865000 (+ 3858us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:42565: BlockingRecv error: recv got EOF from 127.0.97.1:42565 (error 108)
Metrics: {"server-negotiator.queue_time_us":163}
Jun 20 23:36:55 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:36:55 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:36:55.356227 9688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:55.347698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41309 (local address 127.0.97.2:51265)
0620 23:36:55.348179 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:55.348195 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:55.348306 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:55.348651 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:55.348655 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:55.349020 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:55.349202 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:55.349211 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:55.350218 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:55.350222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:55.351266 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:55.351278 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:55.351952 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:55.351956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:55.352107 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:55.352723 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:55.352741 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:55.355872 (+ 3131us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41309: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":261,"threads_started":1}
W20260620 23:36:55.356382 9646 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:36:55.347780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51265 (local address 127.0.97.62:41309)
0620 23:36:55.347948 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:55.347952 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:55.348267 (+ 315us) server_negotiation.cc:407] Connection header received
0620 23:36:55.348400 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:55.348404 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:55.348465 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:55.348549 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:55.349347 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:55.350067 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:55.351417 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:55.351809 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:55.351928 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:55.356087 (+ 4159us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:51265: BlockingRecv error: recv got EOF from 127.0.97.2:51265 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 23:36:55 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:36:55 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:36:55.853714 9689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:55.845361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41309 (local address 127.0.97.3:52039)
0620 23:36:55.845854 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:55.845871 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:55.845979 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:55.846265 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:55.846268 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:55.846605 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:55.846794 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:55.846803 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:55.847856 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:55.847861 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:55.848922 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:55.848936 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:55.849626 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:55.849631 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:55.849759 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:55.850357 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:55.850376 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:55.853366 (+ 2990us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41309: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":260,"threads_started":1}
W20260620 23:36:55.853904 9646 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:36:55.845423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52039 (local address 127.0.97.62:41309)
0620 23:36:55.845587 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:55.845592 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:55.845937 (+ 345us) server_negotiation.cc:407] Connection header received
0620 23:36:55.846080 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:55.846083 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:55.846141 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:55.846227 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:55.846909 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:55.847684 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:55.849061 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:55.849490 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:55.849578 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:55.853654 (+ 4076us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:52039: BlockingRecv error: recv got EOF from 127.0.97.3:52039 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 23:36:55 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:36:55 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:36:55.874684 9690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:55.866361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41309 (local address 127.0.97.1:42267)
0620 23:36:55.866925 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:55.866984 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:55.867143 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:55.867483 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:55.867488 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:55.867915 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:55.868101 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:55.868110 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:55.869074 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:55.869081 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:55.870027 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:55.870039 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:55.870739 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:55.870745 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:55.870918 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:55.871634 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:55.871652 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:55.874420 (+ 2768us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41309: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":335,"threads_started":1}
W20260620 23:36:55.874843 9646 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:36:55.866412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42267 (local address 127.0.97.62:41309)
0620 23:36:55.866637 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:55.866646 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:55.867075 (+ 429us) server_negotiation.cc:407] Connection header received
0620 23:36:55.867282 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:55.867286 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:55.867348 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:55.867451 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:55.868238 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:55.868940 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:55.870167 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:55.870592 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:55.870728 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:55.874631 (+ 3903us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:42267: BlockingRecv error: recv got EOF from 127.0.97.1:42267 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 23:36:56 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:36:56 dist-test-slave-tcgx krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:36:56.366430 9691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:56.357642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41309 (local address 127.0.97.2:53311)
0620 23:36:56.358156 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:56.358177 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:56.358344 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:56.358699 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:56.358704 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:56.359222 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:56.359409 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:56.359418 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:56.360672 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:56.360678 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:56.361592 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:56.361611 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:56.362313 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:56.362319 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:56.362490 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:56.363150 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:36:56.363168 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:56.366045 (+ 2877us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41309: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":253,"threads_started":1}
W20260620 23:36:56.366564 9646 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:36:56.357916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53311 (local address 127.0.97.62:41309)
0620 23:36:56.358134 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:56.358141 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:56.358272 (+ 131us) server_negotiation.cc:407] Connection header received
0620 23:36:56.358483 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:56.358489 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:56.358566 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:56.358675 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:56.359553 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:56.360520 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:56.361742 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:56.362165 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:56.362301 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:56.366302 (+ 4001us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:53311: BlockingRecv error: recv got EOF from 127.0.97.2:53311 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 23:36:56.667728 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 9233
I20260620 23:36:56.680828 9693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:56.669059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56066 (local address 127.0.97.1:44865)
0620 23:36:56.669565 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:56.669572 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:56.669600 (+ 28us) server_negotiation.cc:407] Connection header received
0620 23:36:56.669659 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:56.669665 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:56.669898 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:56.670049 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:56.671815 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:56.672845 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:56.674225 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:56.674775 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:56.674873 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:56.677575 (+ 2702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:56.677583 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:56.677597 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:56.677630 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:56.680320 (+ 2690us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 23:36:56.680595 (+ 275us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56066: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":245,"threads_started":1}
W20260620 23:36:56.680951 9693 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56066: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 23:36:56.681221 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 9233: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:44865: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 23:36:56.681423 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 9233
I20260620 23:36:56.698675 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 9384
I20260620 23:36:56.711354 9694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:56.699899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43494 (local address 127.0.97.2:40637)
0620 23:36:56.700470 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:56.700477 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:56.700509 (+ 32us) server_negotiation.cc:407] Connection header received
0620 23:36:56.700570 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:56.700576 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:56.700905 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:56.701111 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:56.702385 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:56.703593 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:56.704797 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:56.705351 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:56.705447 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:56.708110 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:56.708124 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:56.708139 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:56.708174 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:56.710680 (+ 2506us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 23:36:56.711094 (+ 414us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43494: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":276,"threads_started":1}
W20260620 23:36:56.711473 9694 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43494: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 23:36:56.711483 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 9384: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:40637: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 23:36:56.711665 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 9384
I20260620 23:36:56.728101 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 9515
I20260620 23:36:56.740482 9695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:56.729076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60928 (local address 127.0.97.3:36495)
0620 23:36:56.729679 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:56.729687 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:56.729714 (+ 27us) server_negotiation.cc:407] Connection header received
0620 23:36:56.729780 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:56.729787 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:56.730009 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:56.730178 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:56.731528 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:56.732578 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:56.733880 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:56.734556 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:56.734661 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:56.737396 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:56.737405 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:56.737420 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:36:56.737458 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:56.739986 (+ 2528us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 23:36:56.740252 (+ 266us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60928: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":344,"threads_started":1}
W20260620 23:36:56.740603 9695 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60928: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 23:36:56.740700 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 9515: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:36495: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 23:36:56.740883 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 9515
I20260620 23:36:56.757377 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 9164
I20260620 23:36:56.758822 9226 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:36:56.874136 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 9164
2026-06-20T23:36:56Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21231 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T23:36:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:36:56Z Disabled control of system clock
I20260620 23:36:56.951845 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998525501163-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998525501163-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998525501163-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38485
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:43441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38485
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 23:36:57.285751 9700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:36:57.286055 9700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:36:57.286142 9700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:36:57.293648 9700 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:36:57.293761 9700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:36:57.293798 9700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:36:57.293855 9700 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:36:57.293898 9700 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:36:57.303333 9700 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998525501163-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998525501163-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38485
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998525501163-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:36:57.304632 9700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:36:57.306409 9700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:36:57.318830 9705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:36:57.318902 9700 server_base.cc:1034] running on GCE node
W20260620 23:36:57.319048 9706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:57.318794 9708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:36:57.320114 9700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:36:57.321724 9700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:36:57.323005 9700 hybrid_clock.cc:648] HybridClock initialized: now 1781998617322916 us; error 43 us; skew 500 ppm
I20260620 23:36:57.323360 9700 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:36:57.325591 9700 webserver.cc:460] Webserver started at http://127.0.97.62:39375/ using document root <none> and password file <none>
I20260620 23:36:57.326313 9700 fs_manager.cc:359] Metadata directory not provided
I20260620 23:36:57.326443 9700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:36:57.326781 9700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:36:57.329639 9700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998525501163-388-0/minicluster-data/master-0/data/instance:
uuid: "609af7e0d25b44d7aa3078fa56f7d233"
format_stamp: "Formatted at 2026-06-20 23:36:57 on dist-test-slave-tcgx"
server_key: "ccd0b4623635966642744929f51f7fcb"
server_key_iv: "28e1d2c5347993d2502c31c1682fa596"
server_key_version: "encryptionkey@0"
I20260620 23:36:57.330438 9700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998525501163-388-0/minicluster-data/master-0/wal/instance:
uuid: "609af7e0d25b44d7aa3078fa56f7d233"
format_stamp: "Formatted at 2026-06-20 23:36:57 on dist-test-slave-tcgx"
server_key: "ccd0b4623635966642744929f51f7fcb"
server_key_iv: "28e1d2c5347993d2502c31c1682fa596"
server_key_version: "encryptionkey@0"
I20260620 23:36:57.335912 9700 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 23:36:57.339596 9714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:57.341037 9700 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 23:36:57.341243 9700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998525501163-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998525501163-388-0/minicluster-data/master-0/wal
uuid: "609af7e0d25b44d7aa3078fa56f7d233"
format_stamp: "Formatted at 2026-06-20 23:36:57 on dist-test-slave-tcgx"
server_key: "ccd0b4623635966642744929f51f7fcb"
server_key_iv: "28e1d2c5347993d2502c31c1682fa596"
server_key_version: "encryptionkey@0"
I20260620 23:36:57.341466 9700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998525501163-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998525501163-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998525501163-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:36:57.378314 9700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:36:57.379318 9700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:36:57.379642 9700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:36:57.398639 9700 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38485
I20260620 23:36:57.398672 9765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38485 every 8 connection(s)
I20260620 23:36:57.400136 9700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:36:57.404014 9766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:36:57.404887 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 9700
I20260620 23:36:57.405108 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998525501163-388-0/minicluster-data/master-0/wal/instance
I20260620 23:36:57.405782 388 external_mini_cluster.cc:1428] Setting key e6fa9e481c1fbc4c685e6303df3555e1
I20260620 23:36:57.418243 9766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 609af7e0d25b44d7aa3078fa56f7d233: Bootstrap starting.
I20260620 23:36:57.420145 9769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:57.407981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48658 (local address 127.0.97.62:38485)
0620 23:36:57.409024 (+ 1043us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:57.409038 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:57.409087 (+ 49us) server_negotiation.cc:407] Connection header received
0620 23:36:57.410655 (+ 1568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:57.410677 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:57.411251 (+ 574us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:57.411589 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:57.412272 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:57.413519 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:57.415502 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:57.416333 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:57.416467 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:57.417743 (+ 1276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:57.417751 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:57.417766 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:36:57.418321 (+ 555us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:57.418450 (+ 129us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:36:57.418484 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:57.418579 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:57.418804 (+ 225us) server_negotiation.cc:299] Negotiation successful
0620 23:36:57.419199 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":392,"threads_started":1}
I20260620 23:36:57.421896 9766 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 609af7e0d25b44d7aa3078fa56f7d233: Neither blocks nor log segments found. Creating new log.
I20260620 23:36:57.423066 9766 log.cc:824] T 00000000000000000000000000000000 P 609af7e0d25b44d7aa3078fa56f7d233: Log is configured to *not* fsync() on all Append() calls
I20260620 23:36:57.425716 9766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 609af7e0d25b44d7aa3078fa56f7d233: No bootstrap required, opened a new log
I20260620 23:36:57.433189 9766 raft_consensus.cc:348] T 00000000000000000000000000000000 P 609af7e0d25b44d7aa3078fa56f7d233 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "609af7e0d25b44d7aa3078fa56f7d233" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38485 } }
I20260620 23:36:57.433584 9766 raft_consensus.cc:374] T 00000000000000000000000000000000 P 609af7e0d25b44d7aa3078fa56f7d233 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:36:57.433701 9766 raft_consensus.cc:729] T 00000000000000000000000000000000 P 609af7e0d25b44d7aa3078fa56f7d233 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 609af7e0d25b44d7aa3078fa56f7d233, State: Initialized, Role: FOLLOWER
I20260620 23:36:57.434368 9766 consensus_queue.cc:260] T 00000000000000000000000000000000 P 609af7e0d25b44d7aa3078fa56f7d233 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "609af7e0d25b44d7aa3078fa56f7d233" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38485 } }
I20260620 23:36:57.434614 9766 raft_consensus.cc:388] T 00000000000000000000000000000000 P 609af7e0d25b44d7aa3078fa56f7d233 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:36:57.434808 9766 raft_consensus.cc:482] T 00000000000000000000000000000000 P 609af7e0d25b44d7aa3078fa56f7d233 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:36:57.435019 9766 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 609af7e0d25b44d7aa3078fa56f7d233 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:36:57.437176 9766 raft_consensus.cc:504] T 00000000000000000000000000000000 P 609af7e0d25b44d7aa3078fa56f7d233 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "609af7e0d25b44d7aa3078fa56f7d233" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38485 } }
I20260620 23:36:57.437704 9766 leader_election.cc:302] T 00000000000000000000000000000000 P 609af7e0d25b44d7aa3078fa56f7d233 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 609af7e0d25b44d7aa3078fa56f7d233; no voters:
I20260620 23:36:57.438314 9766 leader_election.cc:288] T 00000000000000000000000000000000 P 609af7e0d25b44d7aa3078fa56f7d233 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:36:57.438460 9771 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 609af7e0d25b44d7aa3078fa56f7d233 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:36:57.439420 9771 raft_consensus.cc:686] T 00000000000000000000000000000000 P 609af7e0d25b44d7aa3078fa56f7d233 [term 1 LEADER]: Becoming Leader. State: Replica: 609af7e0d25b44d7aa3078fa56f7d233, State: Running, Role: LEADER
I20260620 23:36:57.439988 9771 consensus_queue.cc:237] T 00000000000000000000000000000000 P 609af7e0d25b44d7aa3078fa56f7d233 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "609af7e0d25b44d7aa3078fa56f7d233" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38485 } }
I20260620 23:36:57.440604 9766 sys_catalog.cc:564] T 00000000000000000000000000000000 P 609af7e0d25b44d7aa3078fa56f7d233 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:36:57.442811 9773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 609af7e0d25b44d7aa3078fa56f7d233 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 609af7e0d25b44d7aa3078fa56f7d233. Latest consensus state: current_term: 1 leader_uuid: "609af7e0d25b44d7aa3078fa56f7d233" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "609af7e0d25b44d7aa3078fa56f7d233" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38485 } } }
I20260620 23:36:57.443331 9773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 609af7e0d25b44d7aa3078fa56f7d233 [sys.catalog]: This master's current role is: LEADER
I20260620 23:36:57.442803 9772 sys_catalog.cc:455] T 00000000000000000000000000000000 P 609af7e0d25b44d7aa3078fa56f7d233 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "609af7e0d25b44d7aa3078fa56f7d233" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "609af7e0d25b44d7aa3078fa56f7d233" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38485 } } }
I20260620 23:36:57.444583 9772 sys_catalog.cc:458] T 00000000000000000000000000000000 P 609af7e0d25b44d7aa3078fa56f7d233 [sys.catalog]: This master's current role is: LEADER
I20260620 23:36:57.448148 9780 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:36:57.452457 9780 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:36:57.460654 9780 catalog_manager.cc:1347] Generated new cluster ID: df5c1882458a4c5cbba3bc1dd534dd91
I20260620 23:36:57.460788 9780 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:36:57.473878 9780 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:36:57.475274 9780 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:36:57.486150 9780 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 609af7e0d25b44d7aa3078fa56f7d233: Generated new TSK 0
I20260620 23:36:57.487107 9780 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:36:57.494168 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998525501163-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998525501163-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998525501163-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38485
--builtin_ntp_servers=127.0.97.20:43441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 23:36:57.855192 9790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:36:57.855496 9790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:36:57.855588 9790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:36:57.863126 9790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:36:57.863238 9790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:36:57.863343 9790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 23:36:57.872831 9790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998525501163-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998525501163-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38485
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998525501163-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:36:57.874138 9790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:36:57.875883 9790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:36:57.889138 9795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:57.889138 9796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:57.889144 9798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:36:57.890115 9790 server_base.cc:1034] running on GCE node
I20260620 23:36:57.890841 9790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:36:57.891948 9790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:36:57.893182 9790 hybrid_clock.cc:648] HybridClock initialized: now 1781998617893121 us; error 62 us; skew 500 ppm
I20260620 23:36:57.893510 9790 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:36:57.899370 9790 webserver.cc:460] Webserver started at http://127.0.97.1:36993/ using document root <none> and password file <none>
I20260620 23:36:57.900151 9790 fs_manager.cc:359] Metadata directory not provided
I20260620 23:36:57.900290 9790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:36:57.900641 9790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:36:57.903765 9790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998525501163-388-0/minicluster-data/ts-0/data/instance:
uuid: "b5df42b0140c421fa9bd69d6ab7adc51"
format_stamp: "Formatted at 2026-06-20 23:36:57 on dist-test-slave-tcgx"
server_key: "9e122d245db83fd2d5da090fbfca1e19"
server_key_iv: "591cc9a876f427ae62542ea8c55d6fc5"
server_key_version: "encryptionkey@0"
I20260620 23:36:57.904580 9790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998525501163-388-0/minicluster-data/ts-0/wal/instance:
uuid: "b5df42b0140c421fa9bd69d6ab7adc51"
format_stamp: "Formatted at 2026-06-20 23:36:57 on dist-test-slave-tcgx"
server_key: "9e122d245db83fd2d5da090fbfca1e19"
server_key_iv: "591cc9a876f427ae62542ea8c55d6fc5"
server_key_version: "encryptionkey@0"
I20260620 23:36:57.910120 9790 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 23:36:57.914371 9804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:57.915952 9790 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 23:36:57.916152 9790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998525501163-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998525501163-388-0/minicluster-data/ts-0/wal
uuid: "b5df42b0140c421fa9bd69d6ab7adc51"
format_stamp: "Formatted at 2026-06-20 23:36:57 on dist-test-slave-tcgx"
server_key: "9e122d245db83fd2d5da090fbfca1e19"
server_key_iv: "591cc9a876f427ae62542ea8c55d6fc5"
server_key_version: "encryptionkey@0"
I20260620 23:36:57.916388 9790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998525501163-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998525501163-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998525501163-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:36:57.948783 9790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:36:57.949681 9790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:36:57.949966 9790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:36:57.950899 9790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:36:57.952507 9790 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:36:57.952639 9790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:57.952780 9790 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:36:57.952868 9790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:57.986639 9790 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37909
I20260620 23:36:57.986696 9916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37909 every 8 connection(s)
I20260620 23:36:57.988026 9790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
I20260620 23:36:57.993065 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 9790
I20260620 23:36:57.993342 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998525501163-388-0/minicluster-data/ts-0/wal/instance
I20260620 23:36:57.994140 388 external_mini_cluster.cc:1428] Setting key b438070e779215f8fff0232595e03433
I20260620 23:36:57.997577 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998525501163-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998525501163-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998525501163-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38485
--builtin_ntp_servers=127.0.97.20:43441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 23:36:58.003130 9919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:57.990807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50117 (local address 127.0.97.62:38485)
0620 23:36:57.991304 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:57.991310 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:57.992695 (+ 1385us) server_negotiation.cc:407] Connection header received
0620 23:36:57.995259 (+ 2564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:57.995265 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:57.995397 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:57.995510 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:57.996830 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:57.997896 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:57.999763 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:58.000463 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:58.000620 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:58.002088 (+ 1468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:58.002097 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:58.002105 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:36:58.002133 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:58.002185 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:36:58.002205 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:58.002300 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:58.002520 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 23:36:58.002645 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":280,"threads_started":1}
I20260620 23:36:58.004082 9920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:57.991569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38485 (local address 127.0.97.1:50117)
0620 23:36:57.992473 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:57.992514 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:57.994916 (+ 2402us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:57.995825 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:57.995838 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:57.995968 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:57.996653 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:57.996672 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:57.998099 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:57.998103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:57.999597 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:57.999611 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:58.000696 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:58.000702 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:58.000933 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:58.001740 (+ 807us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:36:58.001743 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:58.001787 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:36:58.001875 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:36:58.002305 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:36:58.002312 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:36:58.002318 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 23:36:58.002472 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 23:36:58.002798 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":269,"threads_started":1}
I20260620 23:36:58.005736 9917 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38485
I20260620 23:36:58.006067 9917 heartbeater.cc:464] Registering TS with master...
I20260620 23:36:58.006836 9917 heartbeater.cc:511] Master 127.0.97.62:38485 requested a full tablet report, sending...
I20260620 23:36:58.009037 9731 ts_manager.cc:194] Registered new tserver with Master: b5df42b0140c421fa9bd69d6ab7adc51 (127.0.97.1:37909)
I20260620 23:36:58.010727 9731 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:50117
W20260620 23:36:58.337540 9921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:36:58.337844 9921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:36:58.337988 9921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:36:58.346350 9921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:36:58.346498 9921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:36:58.346726 9921 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 23:36:58.356798 9921 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998525501163-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998525501163-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38485
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998525501163-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:36:58.358136 9921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:36:58.359980 9921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:36:58.373921 9929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:58.374022 9926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:58.373956 9927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:36:58.374670 9921 server_base.cc:1034] running on GCE node
I20260620 23:36:58.375439 9921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:36:58.376528 9921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:36:58.377786 9921 hybrid_clock.cc:648] HybridClock initialized: now 1781998618377727 us; error 55 us; skew 500 ppm
I20260620 23:36:58.378167 9921 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:36:58.380796 9921 webserver.cc:460] Webserver started at http://127.0.97.2:37711/ using document root <none> and password file <none>
I20260620 23:36:58.381695 9921 fs_manager.cc:359] Metadata directory not provided
I20260620 23:36:58.381789 9921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:36:58.382155 9921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:36:58.385337 9921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998525501163-388-0/minicluster-data/ts-1/data/instance:
uuid: "8acbe8e1d32e470faa04419633694698"
format_stamp: "Formatted at 2026-06-20 23:36:58 on dist-test-slave-tcgx"
server_key: "cfdf35e27a939bc79fc9c0c83de7151b"
server_key_iv: "389bfdfcee9d17cb3cd6ea55b54a0f68"
server_key_version: "encryptionkey@0"
I20260620 23:36:58.386173 9921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998525501163-388-0/minicluster-data/ts-1/wal/instance:
uuid: "8acbe8e1d32e470faa04419633694698"
format_stamp: "Formatted at 2026-06-20 23:36:58 on dist-test-slave-tcgx"
server_key: "cfdf35e27a939bc79fc9c0c83de7151b"
server_key_iv: "389bfdfcee9d17cb3cd6ea55b54a0f68"
server_key_version: "encryptionkey@0"
I20260620 23:36:58.391803 9921 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 23:36:58.395536 9935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:58.397183 9921 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 23:36:58.397493 9921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998525501163-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998525501163-388-0/minicluster-data/ts-1/wal
uuid: "8acbe8e1d32e470faa04419633694698"
format_stamp: "Formatted at 2026-06-20 23:36:58 on dist-test-slave-tcgx"
server_key: "cfdf35e27a939bc79fc9c0c83de7151b"
server_key_iv: "389bfdfcee9d17cb3cd6ea55b54a0f68"
server_key_version: "encryptionkey@0"
I20260620 23:36:58.397744 9921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998525501163-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998525501163-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998525501163-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:36:58.415118 9921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:36:58.416056 9921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:36:58.416298 9921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:36:58.417208 9921 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:36:58.418751 9921 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:36:58.418836 9921 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:58.418920 9921 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:36:58.419044 9921 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260620 23:36:58.453460 9921 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36295
I20260620 23:36:58.453495 10047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36295 every 8 connection(s)
I20260620 23:36:58.454779 9921 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
I20260620 23:36:58.462898 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 9921
I20260620 23:36:58.463150 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998525501163-388-0/minicluster-data/ts-1/wal/instance
I20260620 23:36:58.463573 388 external_mini_cluster.cc:1428] Setting key e5f51fc850b9b1edb5e3eae217cd3f31
I20260620 23:36:58.467000 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998525501163-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998525501163-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998525501163-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38485
--builtin_ntp_servers=127.0.97.20:43441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 23:36:58.468261 9919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:58.457307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34539 (local address 127.0.97.62:38485)
0620 23:36:58.457485 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:58.457489 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:58.458902 (+ 1413us) server_negotiation.cc:407] Connection header received
0620 23:36:58.460615 (+ 1713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:58.460619 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:58.460699 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:58.460788 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:58.461893 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:58.462675 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:58.465031 (+ 2356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:58.465465 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:58.465608 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:58.467377 (+ 1769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:58.467382 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:58.467386 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:36:58.467406 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:58.467440 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:36:58.467453 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:58.467540 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:58.467902 (+ 362us) server_negotiation.cc:299] Negotiation successful
0620 23:36:58.467984 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":64}
I20260620 23:36:58.469733 10050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:58.457869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38485 (local address 127.0.97.2:34539)
0620 23:36:58.458739 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:58.458780 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:58.460357 (+ 1577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:58.461044 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:58.461053 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:58.461146 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:58.461738 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:58.461755 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:58.462860 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:58.462865 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:58.464872 (+ 2007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:58.464889 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:58.465695 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:58.465702 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:58.466030 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:58.466887 (+ 857us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:36:58.466893 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:58.466982 (+ 89us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:36:58.467097 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:36:58.467564 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:36:58.467577 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:36:58.467587 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 23:36:58.467779 (+ 192us) client_negotiation.cc:240] Negotiation successful
0620 23:36:58.468254 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":250,"threads_started":1}
I20260620 23:36:58.471222 10048 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38485
I20260620 23:36:58.471554 10048 heartbeater.cc:464] Registering TS with master...
I20260620 23:36:58.472365 10048 heartbeater.cc:511] Master 127.0.97.62:38485 requested a full tablet report, sending...
I20260620 23:36:58.473798 9731 ts_manager.cc:194] Registered new tserver with Master: 8acbe8e1d32e470faa04419633694698 (127.0.97.2:36295)
I20260620 23:36:58.475028 9731 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:34539
W20260620 23:36:58.819947 10051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:36:58.820307 10051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:36:58.820477 10051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:36:58.828101 10051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:36:58.828217 10051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:36:58.828321 10051 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 23:36:58.837893 10051 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998525501163-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998525501163-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38485
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998525501163-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:36:58.839840 10051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:36:58.842175 10051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:36:58.855772 10056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:58.855849 10059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:58.855885 10057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:36:58.856823 10051 server_base.cc:1034] running on GCE node
I20260620 23:36:58.857471 10051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:36:58.858484 10051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:36:58.859702 10051 hybrid_clock.cc:648] HybridClock initialized: now 1781998618859640 us; error 52 us; skew 500 ppm
I20260620 23:36:58.860049 10051 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:36:58.862511 10051 webserver.cc:460] Webserver started at http://127.0.97.3:35791/ using document root <none> and password file <none>
I20260620 23:36:58.863402 10051 fs_manager.cc:359] Metadata directory not provided
I20260620 23:36:58.863494 10051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:36:58.863857 10051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:36:58.866732 10051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998525501163-388-0/minicluster-data/ts-2/data/instance:
uuid: "19e61cefba76452691b2b7320e191b3e"
format_stamp: "Formatted at 2026-06-20 23:36:58 on dist-test-slave-tcgx"
server_key: "be6ade5620e999af2be5b2d1035692cb"
server_key_iv: "6b06671c5af6b465c4baf0df653c18a1"
server_key_version: "encryptionkey@0"
I20260620 23:36:58.867578 10051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998525501163-388-0/minicluster-data/ts-2/wal/instance:
uuid: "19e61cefba76452691b2b7320e191b3e"
format_stamp: "Formatted at 2026-06-20 23:36:58 on dist-test-slave-tcgx"
server_key: "be6ade5620e999af2be5b2d1035692cb"
server_key_iv: "6b06671c5af6b465c4baf0df653c18a1"
server_key_version: "encryptionkey@0"
I20260620 23:36:58.872879 10051 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:36:58.876556 10065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:58.878017 10051 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 23:36:58.878226 10051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998525501163-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998525501163-388-0/minicluster-data/ts-2/wal
uuid: "19e61cefba76452691b2b7320e191b3e"
format_stamp: "Formatted at 2026-06-20 23:36:58 on dist-test-slave-tcgx"
server_key: "be6ade5620e999af2be5b2d1035692cb"
server_key_iv: "6b06671c5af6b465c4baf0df653c18a1"
server_key_version: "encryptionkey@0"
I20260620 23:36:58.878468 10051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998525501163-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998525501163-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998525501163-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:36:58.921918 10051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:36:58.922825 10051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:36:58.923120 10051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:36:58.924140 10051 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:36:58.925706 10051 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:36:58.925799 10051 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:58.925957 10051 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:36:58.926054 10051 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:58.960027 10051 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42767
I20260620 23:36:58.960114 10177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42767 every 8 connection(s)
I20260620 23:36:58.961413 10051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
I20260620 23:36:58.963372 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 10051
I20260620 23:36:58.963565 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998525501163-388-0/minicluster-data/ts-2/wal/instance
I20260620 23:36:58.963989 388 external_mini_cluster.cc:1428] Setting key 9440f47c0ac3b38501cf98fb297cb8e1
I20260620 23:36:58.975080 9919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:58.964429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46819 (local address 127.0.97.62:38485)
0620 23:36:58.964632 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:58.964636 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:58.966551 (+ 1915us) server_negotiation.cc:407] Connection header received
0620 23:36:58.968603 (+ 2052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:58.968607 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:58.968686 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:58.968782 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:58.969928 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:58.970782 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:58.972389 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:58.972816 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:58.972921 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:58.974269 (+ 1348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:58.974274 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:58.974278 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:36:58.974297 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:58.974330 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:36:58.974342 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:58.974417 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:58.974613 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 23:36:58.974691 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 23:36:58.976058 10180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:58.965155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38485 (local address 127.0.97.3:46819)
0620 23:36:58.966402 (+ 1247us) negotiation.cc:107] Waiting for socket to connect
0620 23:36:58.966442 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 23:36:58.968310 (+ 1868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:36:58.969063 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:36:58.969072 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:36:58.969177 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:36:58.969763 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:58.969780 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:58.970918 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:58.970922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:58.972251 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:36:58.972263 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:36:58.973010 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:36:58.973015 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:36:58.973225 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:58.973961 (+ 736us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:36:58.973964 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:36:58.974008 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:36:58.974090 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:36:58.974427 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:36:58.974434 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:36:58.974440 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 23:36:58.974600 (+ 160us) client_negotiation.cc:240] Negotiation successful
0620 23:36:58.974897 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":377,"threads_started":1}
I20260620 23:36:58.977639 10178 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38485
I20260620 23:36:58.977957 10178 heartbeater.cc:464] Registering TS with master...
I20260620 23:36:58.978807 10178 heartbeater.cc:511] Master 127.0.97.62:38485 requested a full tablet report, sending...
I20260620 23:36:58.980338 9731 ts_manager.cc:194] Registered new tserver with Master: 19e61cefba76452691b2b7320e191b3e (127.0.97.3:42767)
I20260620 23:36:58.981084 9731 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:46819
I20260620 23:36:58.992460 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 23:36:58.999104 9919 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:36:58.997249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48662 (local address 127.0.97.62:38485)
0620 23:36:58.997491 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:58.997501 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:58.997899 (+ 398us) server_negotiation.cc:407] Connection header received
0620 23:36:58.998022 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:58.998026 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:58.998139 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:58.998223 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:58.998773 (+ 550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48662: BlockingRecv error: recv got EOF from 127.0.0.1:48662 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 23:36:58.999655 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 9790
I20260620 23:36:59.008117 10187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:59.000876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55720 (local address 127.0.97.1:37909)
0620 23:36:59.001453 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:59.001463 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:59.001503 (+ 40us) server_negotiation.cc:407] Connection header received
0620 23:36:59.001562 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:59.001568 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:59.001780 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:59.001959 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:59.002377 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:59.003473 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:59.004757 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:59.005335 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:59.005476 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:59.006615 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:59.006623 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:59.006637 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:36:59.007298 (+ 661us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:59.007346 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:36:59.007372 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:59.007518 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:59.007622 (+ 104us) server_negotiation.cc:299] Negotiation successful
0620 23:36:59.007760 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":246,"threads_started":1}
I20260620 23:36:59.009377 9912 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:36:59.091241 9917 heartbeater.cc:503] Master 127.0.97.62:38485 was elected leader, sending a full tablet report...
I20260620 23:36:59.092449 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 9790
W20260620 23:36:59.108811 9717 connection.cc:535] server connection from 127.0.97.1:50117 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 23:36:59.109169 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 9921
I20260620 23:36:59.117846 10189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:59.110142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58026 (local address 127.0.97.2:36295)
0620 23:36:59.110727 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:59.110735 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:59.110772 (+ 37us) server_negotiation.cc:407] Connection header received
0620 23:36:59.110844 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:59.110850 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:59.111230 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:59.111388 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:59.111812 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:59.112890 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:59.114190 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:59.114859 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:59.115080 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:59.116109 (+ 1029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:59.116118 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:59.116134 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:36:59.116809 (+ 675us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:59.116866 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:36:59.116887 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:59.116986 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:59.117240 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 23:36:59.117431 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":308,"threads_started":1}
I20260620 23:36:59.119284 10043 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:36:59.203866 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 9921
I20260620 23:36:59.220070 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 10051
I20260620 23:36:59.228991 10191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:59.221112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58952 (local address 127.0.97.3:42767)
0620 23:36:59.221745 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:59.221760 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:59.221790 (+ 30us) server_negotiation.cc:407] Connection header received
0620 23:36:59.221850 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:59.221857 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:59.222093 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:59.222382 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:59.222842 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:59.223968 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:59.225303 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:36:59.225892 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:36:59.226034 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:36:59.227338 (+ 1304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:59.227348 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:59.227363 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:36:59.228058 (+ 695us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:59.228118 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:36:59.228141 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:59.228299 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:59.228430 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 23:36:59.228598 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":328,"threads_started":1}
I20260620 23:36:59.230352 10173 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:36:59.313447 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 10051
I20260620 23:36:59.329612 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 9700
I20260620 23:36:59.331138 9761 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:36:59.434921 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 9700
2026-06-20T23:36:59Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2555 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T23:36:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:36:59Z Disabled control of system clock
I20260620 23:36:59.509191 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998525501163-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998525501163-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998525501163-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46547
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:42423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46547
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 23:36:59.853824 10197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:36:59.854130 10197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:36:59.854334 10197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:36:59.861822 10197 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:36:59.861949 10197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:36:59.862046 10197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:36:59.862160 10197 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:36:59.862233 10197 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:36:59.872056 10197 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998525501163-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998525501163-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46547
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998525501163-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:36:59.873476 10197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:36:59.875454 10197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:36:59.888001 10202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:59.887912 10203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:36:59.887971 10205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:36:59.889027 10197 server_base.cc:1034] running on GCE node
I20260620 23:36:59.889911 10197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:36:59.891502 10197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:36:59.892778 10197 hybrid_clock.cc:648] HybridClock initialized: now 1781998619892735 us; error 38 us; skew 500 ppm
I20260620 23:36:59.893185 10197 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:36:59.895815 10197 webserver.cc:460] Webserver started at http://127.0.97.62:42317/ using document root <none> and password file <none>
I20260620 23:36:59.896555 10197 fs_manager.cc:359] Metadata directory not provided
I20260620 23:36:59.896689 10197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:36:59.897029 10197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:36:59.900127 10197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998525501163-388-0/minicluster-data/master-0/data/instance:
uuid: "56f86bfeb7eb4b448c8190b8b087ec3a"
format_stamp: "Formatted at 2026-06-20 23:36:59 on dist-test-slave-tcgx"
server_key: "985e3cdf3957a004bba15587946c5781"
server_key_iv: "e825cdfcc0f0ce8998ff25f4c65072bc"
server_key_version: "encryptionkey@0"
I20260620 23:36:59.900979 10197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998525501163-388-0/minicluster-data/master-0/wal/instance:
uuid: "56f86bfeb7eb4b448c8190b8b087ec3a"
format_stamp: "Formatted at 2026-06-20 23:36:59 on dist-test-slave-tcgx"
server_key: "985e3cdf3957a004bba15587946c5781"
server_key_iv: "e825cdfcc0f0ce8998ff25f4c65072bc"
server_key_version: "encryptionkey@0"
I20260620 23:36:59.906452 10197 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 23:36:59.909989 10211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:36:59.911507 10197 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 23:36:59.911691 10197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998525501163-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998525501163-388-0/minicluster-data/master-0/wal
uuid: "56f86bfeb7eb4b448c8190b8b087ec3a"
format_stamp: "Formatted at 2026-06-20 23:36:59 on dist-test-slave-tcgx"
server_key: "985e3cdf3957a004bba15587946c5781"
server_key_iv: "e825cdfcc0f0ce8998ff25f4c65072bc"
server_key_version: "encryptionkey@0"
I20260620 23:36:59.911918 10197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998525501163-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998525501163-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998525501163-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:36:59.917896 10197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:36:59.918681 10197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:36:59.919082 10197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:36:59.937850 10197 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46547
I20260620 23:36:59.937873 10262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46547 every 8 connection(s)
I20260620 23:36:59.939352 10197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:36:59.943316 10263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:36:59.945102 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 10197
I20260620 23:36:59.945297 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998525501163-388-0/minicluster-data/master-0/wal/instance
I20260620 23:36:59.945924 388 external_mini_cluster.cc:1428] Setting key b27416f5137d8a2e918b7fadbe467dab
I20260620 23:36:59.955739 10266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:36:59.948087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39632 (local address 127.0.97.62:46547)
0620 23:36:59.949247 (+ 1160us) server_negotiation.cc:206] Beginning negotiation
0620 23:36:59.949263 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 23:36:59.949316 (+ 53us) server_negotiation.cc:407] Connection header received
0620 23:36:59.951578 (+ 2262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:36:59.951607 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:36:59.952184 (+ 577us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:36:59.952649 (+ 465us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:36:59.953688 (+ 1039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:36:59.953697 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:36:59.953714 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:36:59.953731 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:36:59.953810 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:36:59.953830 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:36:59.954071 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0620 23:36:59.954213 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 23:36:59.954502 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":349,"threads_started":1}
I20260620 23:36:59.956802 10263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56f86bfeb7eb4b448c8190b8b087ec3a: Bootstrap starting.
I20260620 23:36:59.960031 10263 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56f86bfeb7eb4b448c8190b8b087ec3a: Neither blocks nor log segments found. Creating new log.
I20260620 23:36:59.961040 10263 log.cc:824] T 00000000000000000000000000000000 P 56f86bfeb7eb4b448c8190b8b087ec3a: Log is configured to *not* fsync() on all Append() calls
I20260620 23:36:59.963945 10263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56f86bfeb7eb4b448c8190b8b087ec3a: No bootstrap required, opened a new log
I20260620 23:36:59.971776 10263 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56f86bfeb7eb4b448c8190b8b087ec3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56f86bfeb7eb4b448c8190b8b087ec3a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46547 } }
I20260620 23:36:59.972136 10263 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56f86bfeb7eb4b448c8190b8b087ec3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:36:59.972209 10263 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56f86bfeb7eb4b448c8190b8b087ec3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56f86bfeb7eb4b448c8190b8b087ec3a, State: Initialized, Role: FOLLOWER
I20260620 23:36:59.972740 10263 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56f86bfeb7eb4b448c8190b8b087ec3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56f86bfeb7eb4b448c8190b8b087ec3a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46547 } }
I20260620 23:36:59.972939 10263 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56f86bfeb7eb4b448c8190b8b087ec3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:36:59.973026 10263 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56f86bfeb7eb4b448c8190b8b087ec3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:36:59.973146 10263 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56f86bfeb7eb4b448c8190b8b087ec3a [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:36:59.975378 10263 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56f86bfeb7eb4b448c8190b8b087ec3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56f86bfeb7eb4b448c8190b8b087ec3a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46547 } }
I20260620 23:36:59.975919 10263 leader_election.cc:302] T 00000000000000000000000000000000 P 56f86bfeb7eb4b448c8190b8b087ec3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56f86bfeb7eb4b448c8190b8b087ec3a; no voters:
I20260620 23:36:59.976648 10263 leader_election.cc:288] T 00000000000000000000000000000000 P 56f86bfeb7eb4b448c8190b8b087ec3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:36:59.976775 10268 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56f86bfeb7eb4b448c8190b8b087ec3a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:36:59.977459 10268 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56f86bfeb7eb4b448c8190b8b087ec3a [term 1 LEADER]: Becoming Leader. State: Replica: 56f86bfeb7eb4b448c8190b8b087ec3a, State: Running, Role: LEADER
I20260620 23:36:59.978098 10268 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56f86bfeb7eb4b448c8190b8b087ec3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56f86bfeb7eb4b448c8190b8b087ec3a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46547 } }
I20260620 23:36:59.979115 10263 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56f86bfeb7eb4b448c8190b8b087ec3a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:36:59.980998 10270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56f86bfeb7eb4b448c8190b8b087ec3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56f86bfeb7eb4b448c8190b8b087ec3a. Latest consensus state: current_term: 1 leader_uuid: "56f86bfeb7eb4b448c8190b8b087ec3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56f86bfeb7eb4b448c8190b8b087ec3a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46547 } } }
I20260620 23:36:59.981372 10270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56f86bfeb7eb4b448c8190b8b087ec3a [sys.catalog]: This master's current role is: LEADER
I20260620 23:36:59.986646 10269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56f86bfeb7eb4b448c8190b8b087ec3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56f86bfeb7eb4b448c8190b8b087ec3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56f86bfeb7eb4b448c8190b8b087ec3a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46547 } } }
I20260620 23:36:59.987663 10269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56f86bfeb7eb4b448c8190b8b087ec3a [sys.catalog]: This master's current role is: LEADER
I20260620 23:36:59.988416 10276 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:36:59.995365 10276 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:37:00.004390 10276 catalog_manager.cc:1347] Generated new cluster ID: 913a3927965a43058ac123c6c500d316
I20260620 23:37:00.004515 10276 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:37:00.023039 10276 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:37:00.023546 10276 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:37:00.031296 10276 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56f86bfeb7eb4b448c8190b8b087ec3a: Generated new TSK 0
I20260620 23:37:00.032325 10276 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:37:00.038892 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998525501163-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998525501163-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998525501163-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46547
--builtin_ntp_servers=127.0.97.20:42423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 23:37:00.374572 10287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:37:00.374892 10287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:37:00.375077 10287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:37:00.382916 10287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:37:00.383062 10287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:37:00.383241 10287 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 23:37:00.392671 10287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998525501163-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998525501163-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46547
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998525501163-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:37:00.394043 10287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:37:00.395998 10287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:37:00.409817 10292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:37:00.409786 10293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:37:00.409914 10295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:37:00.410620 10287 server_base.cc:1034] running on GCE node
I20260620 23:37:00.411393 10287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:37:00.412408 10287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:37:00.413640 10287 hybrid_clock.cc:648] HybridClock initialized: now 1781998620413601 us; error 56 us; skew 500 ppm
I20260620 23:37:00.413987 10287 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:37:00.416594 10287 webserver.cc:460] Webserver started at http://127.0.97.1:32965/ using document root <none> and password file <none>
I20260620 23:37:00.417368 10287 fs_manager.cc:359] Metadata directory not provided
I20260620 23:37:00.417495 10287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:37:00.417850 10287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:37:00.422497 10287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998525501163-388-0/minicluster-data/ts-0/data/instance:
uuid: "29d2b385a564400ba66b3766a7e3ff3a"
format_stamp: "Formatted at 2026-06-20 23:37:00 on dist-test-slave-tcgx"
server_key: "731d5bbea6812c105bd3b496aee68bac"
server_key_iv: "cbb5e377b3b0cacb6018c742b4a9a437"
server_key_version: "encryptionkey@0"
I20260620 23:37:00.423457 10287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998525501163-388-0/minicluster-data/ts-0/wal/instance:
uuid: "29d2b385a564400ba66b3766a7e3ff3a"
format_stamp: "Formatted at 2026-06-20 23:37:00 on dist-test-slave-tcgx"
server_key: "731d5bbea6812c105bd3b496aee68bac"
server_key_iv: "cbb5e377b3b0cacb6018c742b4a9a437"
server_key_version: "encryptionkey@0"
I20260620 23:37:00.428762 10287 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260620 23:37:00.432351 10301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:37:00.433691 10287 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:37:00.433889 10287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998525501163-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998525501163-388-0/minicluster-data/ts-0/wal
uuid: "29d2b385a564400ba66b3766a7e3ff3a"
format_stamp: "Formatted at 2026-06-20 23:37:00 on dist-test-slave-tcgx"
server_key: "731d5bbea6812c105bd3b496aee68bac"
server_key_iv: "cbb5e377b3b0cacb6018c742b4a9a437"
server_key_version: "encryptionkey@0"
I20260620 23:37:00.434134 10287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998525501163-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998525501163-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998525501163-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:37:00.439937 10287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:37:00.440718 10287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:37:00.440999 10287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:37:00.441900 10287 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:37:00.443321 10287 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:37:00.443439 10287 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:37:00.443578 10287 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:37:00.443665 10287 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:37:00.477005 10287 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42231
I20260620 23:37:00.477059 10413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42231 every 8 connection(s)
I20260620 23:37:00.478389 10287 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
I20260620 23:37:00.481709 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 10287
I20260620 23:37:00.481913 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998525501163-388-0/minicluster-data/ts-0/wal/instance
I20260620 23:37:00.482761 388 external_mini_cluster.cc:1428] Setting key 593771948cab063a71f99ebc84cca186
I20260620 23:37:00.485910 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998525501163-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998525501163-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998525501163-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46547
--builtin_ntp_servers=127.0.97.20:42423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 23:37:00.487782 10416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:37:00.481099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54193 (local address 127.0.97.62:46547)
0620 23:37:00.481529 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0620 23:37:00.481535 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:37:00.483149 (+ 1614us) server_negotiation.cc:407] Connection header received
0620 23:37:00.485258 (+ 2109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:37:00.485262 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:37:00.485308 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:37:00.485387 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:37:00.487047 (+ 1660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:37:00.487051 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:37:00.487055 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:37:00.487070 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:37:00.487108 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:37:00.487119 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:37:00.487206 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 23:37:00.487454 (+ 248us) server_negotiation.cc:299] Negotiation successful
0620 23:37:00.487501 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":204,"threads_started":1}
I20260620 23:37:00.489410 10417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:37:00.481729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46547 (local address 127.0.97.1:54193)
0620 23:37:00.482968 (+ 1239us) negotiation.cc:107] Waiting for socket to connect
0620 23:37:00.483028 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 23:37:00.484934 (+ 1906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:37:00.485678 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:37:00.485691 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:37:00.485823 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:37:00.486668 (+ 845us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:37:00.486674 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:37:00.486728 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:37:00.486848 (+ 120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:37:00.487217 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:37:00.487230 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:37:00.487243 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 23:37:00.487538 (+ 295us) client_negotiation.cc:240] Negotiation successful
0620 23:37:00.487815 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":345,"threads_started":1}
I20260620 23:37:00.490906 10414 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46547
I20260620 23:37:00.491247 10414 heartbeater.cc:464] Registering TS with master...
I20260620 23:37:00.492008 10414 heartbeater.cc:511] Master 127.0.97.62:46547 requested a full tablet report, sending...
I20260620 23:37:00.494472 10228 ts_manager.cc:194] Registered new tserver with Master: 29d2b385a564400ba66b3766a7e3ff3a (127.0.97.1:42231)
W20260620 23:37:00.817871 10418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:37:00.818169 10418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:37:00.818259 10418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:37:00.825611 10418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:37:00.825732 10418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:37:00.825842 10418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 23:37:00.834916 10418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998525501163-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998525501163-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46547
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998525501163-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:37:00.836223 10418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:37:00.838169 10418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:37:00.852705 10426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:37:00.852790 10423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:37:00.852816 10424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:37:00.853943 10418 server_base.cc:1034] running on GCE node
I20260620 23:37:00.854648 10418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:37:00.855803 10418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:37:00.857035 10418 hybrid_clock.cc:648] HybridClock initialized: now 1781998620856965 us; error 65 us; skew 500 ppm
I20260620 23:37:00.857420 10418 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:37:00.859845 10418 webserver.cc:460] Webserver started at http://127.0.97.2:36589/ using document root <none> and password file <none>
I20260620 23:37:00.860566 10418 fs_manager.cc:359] Metadata directory not provided
I20260620 23:37:00.860682 10418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:37:00.861014 10418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:37:00.864004 10418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998525501163-388-0/minicluster-data/ts-1/data/instance:
uuid: "a1b8ffba4e414ae0844b63ce7447cc51"
format_stamp: "Formatted at 2026-06-20 23:37:00 on dist-test-slave-tcgx"
server_key: "9dd15cff5cf1af9750566aa04823cfe3"
server_key_iv: "71551e01232dda6dad73891f42d6a17c"
server_key_version: "encryptionkey@0"
I20260620 23:37:00.864801 10418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998525501163-388-0/minicluster-data/ts-1/wal/instance:
uuid: "a1b8ffba4e414ae0844b63ce7447cc51"
format_stamp: "Formatted at 2026-06-20 23:37:00 on dist-test-slave-tcgx"
server_key: "9dd15cff5cf1af9750566aa04823cfe3"
server_key_iv: "71551e01232dda6dad73891f42d6a17c"
server_key_version: "encryptionkey@0"
I20260620 23:37:00.869884 10418 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260620 23:37:00.873557 10432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:37:00.875054 10418 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:37:00.875270 10418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998525501163-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998525501163-388-0/minicluster-data/ts-1/wal
uuid: "a1b8ffba4e414ae0844b63ce7447cc51"
format_stamp: "Formatted at 2026-06-20 23:37:00 on dist-test-slave-tcgx"
server_key: "9dd15cff5cf1af9750566aa04823cfe3"
server_key_iv: "71551e01232dda6dad73891f42d6a17c"
server_key_version: "encryptionkey@0"
I20260620 23:37:00.875543 10418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998525501163-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998525501163-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998525501163-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:37:00.881568 10418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:37:00.882516 10418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:37:00.882820 10418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:37:00.883908 10418 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:37:00.885398 10418 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:37:00.885536 10418 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:37:00.885685 10418 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:37:00.885782 10418 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:37:00.920010 10418 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:32985
I20260620 23:37:00.920053 10544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:32985 every 8 connection(s)
I20260620 23:37:00.921469 10418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
I20260620 23:37:00.929978 10416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:37:00.924138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43553 (local address 127.0.97.62:46547)
0620 23:37:00.924363 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 23:37:00.924369 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:37:00.925916 (+ 1547us) server_negotiation.cc:407] Connection header received
0620 23:37:00.927648 (+ 1732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:37:00.927653 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:37:00.927741 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:37:00.927852 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:37:00.929218 (+ 1366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:37:00.929224 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:37:00.929238 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:37:00.929265 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:37:00.929296 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:37:00.929307 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:37:00.929430 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 23:37:00.929630 (+ 200us) server_negotiation.cc:299] Negotiation successful
0620 23:37:00.929700 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 23:37:00.931020 10547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:37:00.924817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46547 (local address 127.0.97.2:43553)
0620 23:37:00.925756 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0620 23:37:00.925797 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 23:37:00.927378 (+ 1581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:37:00.928083 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:37:00.928092 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:37:00.928187 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:37:00.928910 (+ 723us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:37:00.928912 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:37:00.928951 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:37:00.929039 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:37:00.929401 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:37:00.929412 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:37:00.929419 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 23:37:00.929705 (+ 286us) client_negotiation.cc:240] Negotiation successful
0620 23:37:00.929996 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":310,"threads_started":1}
I20260620 23:37:00.931365 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 10418
I20260620 23:37:00.931596 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998525501163-388-0/minicluster-data/ts-1/wal/instance
I20260620 23:37:00.932152 388 external_mini_cluster.cc:1428] Setting key b7fb76d576db85bd7a7c408a6209e5c9
I20260620 23:37:00.932615 10545 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46547
I20260620 23:37:00.932909 10545 heartbeater.cc:464] Registering TS with master...
I20260620 23:37:00.933561 10545 heartbeater.cc:511] Master 127.0.97.62:46547 requested a full tablet report, sending...
I20260620 23:37:00.935042 10228 ts_manager.cc:194] Registered new tserver with Master: a1b8ffba4e414ae0844b63ce7447cc51 (127.0.97.2:32985)
I20260620 23:37:00.935241 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998525501163-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998525501163-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998525501163-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46547
--builtin_ntp_servers=127.0.97.20:42423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 23:37:01.266261 10548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:37:01.266551 10548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:37:01.266707 10548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:37:01.273926 10548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:37:01.274032 10548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:37:01.274214 10548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 23:37:01.283558 10548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998525501163-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998525501163-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46547
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998525501163-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:37:01.284884 10548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:37:01.286661 10548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:37:01.299541 10553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:37:01.299604 10556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:37:01.299645 10554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:37:01.300537 10548 server_base.cc:1034] running on GCE node
I20260620 23:37:01.301183 10548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:37:01.302140 10548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:37:01.303372 10548 hybrid_clock.cc:648] HybridClock initialized: now 1781998621303322 us; error 40 us; skew 500 ppm
I20260620 23:37:01.303718 10548 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:37:01.306017 10548 webserver.cc:460] Webserver started at http://127.0.97.3:43985/ using document root <none> and password file <none>
I20260620 23:37:01.306734 10548 fs_manager.cc:359] Metadata directory not provided
I20260620 23:37:01.306865 10548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:37:01.307200 10548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:37:01.310122 10548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998525501163-388-0/minicluster-data/ts-2/data/instance:
uuid: "3c8de859e8854864807624e179c4b4ea"
format_stamp: "Formatted at 2026-06-20 23:37:01 on dist-test-slave-tcgx"
server_key: "08b0c581c9241744ea73dd4c5887f2ee"
server_key_iv: "f1949732b76a5b344e3bea8459d15ec6"
server_key_version: "encryptionkey@0"
I20260620 23:37:01.310917 10548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998525501163-388-0/minicluster-data/ts-2/wal/instance:
uuid: "3c8de859e8854864807624e179c4b4ea"
format_stamp: "Formatted at 2026-06-20 23:37:01 on dist-test-slave-tcgx"
server_key: "08b0c581c9241744ea73dd4c5887f2ee"
server_key_iv: "f1949732b76a5b344e3bea8459d15ec6"
server_key_version: "encryptionkey@0"
I20260620 23:37:01.316114 10548 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 23:37:01.319752 10562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:37:01.321193 10548 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:37:01.321385 10548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998525501163-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998525501163-388-0/minicluster-data/ts-2/wal
uuid: "3c8de859e8854864807624e179c4b4ea"
format_stamp: "Formatted at 2026-06-20 23:37:01 on dist-test-slave-tcgx"
server_key: "08b0c581c9241744ea73dd4c5887f2ee"
server_key_iv: "f1949732b76a5b344e3bea8459d15ec6"
server_key_version: "encryptionkey@0"
I20260620 23:37:01.321621 10548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998525501163-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998525501163-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998525501163-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:37:01.327370 10548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:37:01.328210 10548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:37:01.328471 10548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:37:01.329351 10548 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:37:01.330686 10548 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:37:01.330819 10548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:37:01.330976 10548 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:37:01.331071 10548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260620 23:37:01.365918 10548 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35425
I20260620 23:37:01.365949 10674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35425 every 8 connection(s)
I20260620 23:37:01.367568 10548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
I20260620 23:37:01.368837 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 10548
I20260620 23:37:01.369000 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998525501163-388-0/minicluster-data/ts-2/wal/instance
I20260620 23:37:01.369565 388 external_mini_cluster.cc:1428] Setting key 229aefabe30e3d6ec059f76672add8c4
I20260620 23:37:01.376549 10416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:37:01.370483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48039 (local address 127.0.97.62:46547)
0620 23:37:01.371095 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0620 23:37:01.371101 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:37:01.372387 (+ 1286us) server_negotiation.cc:407] Connection header received
0620 23:37:01.374160 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:37:01.374165 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:37:01.374231 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:37:01.374320 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:37:01.375655 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:37:01.375661 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:37:01.375666 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:37:01.375685 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:37:01.375729 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:37:01.375742 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:37:01.375874 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 23:37:01.376162 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 23:37:01.376234 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485}
I20260620 23:37:01.377483 10677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:37:01.371180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46547 (local address 127.0.97.3:48039)
0620 23:37:01.372236 (+ 1056us) negotiation.cc:107] Waiting for socket to connect
0620 23:37:01.372277 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 23:37:01.373901 (+ 1624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:37:01.374543 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:37:01.374551 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:37:01.374646 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:37:01.375340 (+ 694us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:37:01.375343 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:37:01.375382 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:37:01.375467 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:37:01.375851 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:37:01.375862 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:37:01.375870 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 23:37:01.376266 (+ 396us) client_negotiation.cc:240] Negotiation successful
0620 23:37:01.376531 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":370,"threads_started":1}
I20260620 23:37:01.378790 10675 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46547
I20260620 23:37:01.379145 10675 heartbeater.cc:464] Registering TS with master...
I20260620 23:37:01.379837 10675 heartbeater.cc:511] Master 127.0.97.62:46547 requested a full tablet report, sending...
I20260620 23:37:01.381233 10228 ts_manager.cc:194] Registered new tserver with Master: 3c8de859e8854864807624e179c4b4ea (127.0.97.3:35425)
I20260620 23:37:01.385016 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 23:37:01.391254 10416 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:37:01.389600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39634 (local address 127.0.97.62:46547)
0620 23:37:01.389762 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 23:37:01.389767 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:37:01.390221 (+ 454us) server_negotiation.cc:407] Connection header received
0620 23:37:01.390438 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:37:01.390441 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:37:01.390492 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:37:01.390594 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:37:01.391051 (+ 457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39634: BlockingRecv error: recv got EOF from 127.0.0.1:39634 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 23:37:01.392122 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 10287
I20260620 23:37:01.395517 10684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:37:01.393062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59942 (local address 127.0.97.1:42231)
0620 23:37:01.393612 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0620 23:37:01.393620 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:37:01.393646 (+ 26us) server_negotiation.cc:407] Connection header received
0620 23:37:01.393703 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:37:01.393709 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:37:01.393919 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:37:01.394048 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:37:01.394892 (+ 844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:37:01.394898 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:37:01.394910 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:37:01.394928 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:37:01.395018 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:37:01.395034 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:37:01.395141 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 23:37:01.395222 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 23:37:01.395298 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":263,"threads_started":1}
I20260620 23:37:01.396842 10409 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:37:01.477501 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 10287
I20260620 23:37:01.491698 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 10418
I20260620 23:37:01.495465 10686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:37:01.492636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46300 (local address 127.0.97.2:32985)
0620 23:37:01.493187 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0620 23:37:01.493194 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:37:01.493222 (+ 28us) server_negotiation.cc:407] Connection header received
0620 23:37:01.493283 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:37:01.493289 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:37:01.493510 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:37:01.493689 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:37:01.494733 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:37:01.494740 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:37:01.494753 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:37:01.494770 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:37:01.494809 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:37:01.494823 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:37:01.494998 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 23:37:01.495111 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 23:37:01.495215 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":262,"threads_started":1}
I20260620 23:37:01.496786 10540 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:37:01.580336 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 10418
I20260620 23:37:01.596736 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 10548
I20260620 23:37:01.600476 10688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:37:01.597677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42226 (local address 127.0.97.3:35425)
0620 23:37:01.598340 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0620 23:37:01.598347 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:37:01.598376 (+ 29us) server_negotiation.cc:407] Connection header received
0620 23:37:01.598435 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:37:01.598441 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:37:01.598676 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:37:01.598809 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:37:01.599777 (+ 968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:37:01.599784 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:37:01.599796 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:37:01.599811 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:37:01.599852 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:37:01.599866 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:37:01.599997 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 23:37:01.600051 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 23:37:01.600168 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":352,"threads_started":1}
I20260620 23:37:01.601778 10670 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:37:01.684106 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 10548
I20260620 23:37:01.700380 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 10197
I20260620 23:37:01.701615 10258 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:37:01.804390 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 10197
2026-06-20T23:37:01Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2368 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:37:01 dist-test-slave-tcgx krb5kdc[10693](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:37:01 dist-test-slave-tcgx krb5kdc[10693](info): set up 2 sockets
Jun 20 23:37:01 dist-test-slave-tcgx krb5kdc[10693](info): commencing operation
krb5kdc: starting...
W20260620 23:37:02.907975 388 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/krb5kdc/test-user.keytab.
Jun 20 23:37:03 dist-test-slave-tcgx krb5kdc[10693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:37:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:37:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 23:37:03.142058 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37361
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37361
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:37:03.471484 10706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:37:03.471781 10706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:37:03.471868 10706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:37:03.479086 10706 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:37:03.479205 10706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:37:03.479250 10706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:37:03.479308 10706 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:37:03.479346 10706 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:37:03.489378 10706 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37361
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:37:03.490710 10706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:37:03.492027 10706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:37:03.503914 10711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:37:03.503852 10714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:37:03.503897 10712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:37:03.505357 10706 server_base.cc:1034] running on GCE node
I20260620 23:37:03.506330 10706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:37:03.508641 10706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:37:03.509953 10706 hybrid_clock.cc:648] HybridClock initialized: now 1781998623509899 us; error 45 us; skew 500 ppm
I20260620 23:37:03.510485 10706 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:37:03 dist-test-slave-tcgx krb5kdc[10693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998623, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:37:03.518006 10706 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 23:37:03.519659 10706 webserver.cc:460] Webserver started at http://127.0.97.62:42165/ using document root <none> and password file <none>
I20260620 23:37:03.520362 10706 fs_manager.cc:359] Metadata directory not provided
I20260620 23:37:03.520465 10706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:37:03.520716 10706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:37:03.523782 10706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/minicluster-data/master-0/data/instance:
uuid: "61963474ebdd4c8ea0624c31d44bb43b"
format_stamp: "Formatted at 2026-06-20 23:37:03 on dist-test-slave-tcgx"
server_key: "d7949e51aacfd556e5dd7ce29a7cf15a"
server_key_iv: "2308795080c020a45b7b76f4a6ae7dca"
server_key_version: "encryptionkey@0"
I20260620 23:37:03.524499 10706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/minicluster-data/master-0/wal/instance:
uuid: "61963474ebdd4c8ea0624c31d44bb43b"
format_stamp: "Formatted at 2026-06-20 23:37:03 on dist-test-slave-tcgx"
server_key: "d7949e51aacfd556e5dd7ce29a7cf15a"
server_key_iv: "2308795080c020a45b7b76f4a6ae7dca"
server_key_version: "encryptionkey@0"
I20260620 23:37:03.529734 10706 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 23:37:03.533475 10721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:37:03.535116 10706 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260620 23:37:03.535312 10706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/minicluster-data/master-0/wal
uuid: "61963474ebdd4c8ea0624c31d44bb43b"
format_stamp: "Formatted at 2026-06-20 23:37:03 on dist-test-slave-tcgx"
server_key: "d7949e51aacfd556e5dd7ce29a7cf15a"
server_key_iv: "2308795080c020a45b7b76f4a6ae7dca"
server_key_version: "encryptionkey@0"
I20260620 23:37:03.535516 10706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:37:03.554010 10706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:37:03.558370 10706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:37:03.558681 10706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:37:03.577890 10706 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37361
I20260620 23:37:03.577912 10772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37361 every 8 connection(s)
I20260620 23:37:03.579464 10706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:37:03.583302 10773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:37:03.585238 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 10706
I20260620 23:37:03.585448 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/minicluster-data/master-0/wal/instance
I20260620 23:37:03.586257 388 external_mini_cluster.cc:1428] Setting key fdbeb47b80e5ff7ccff756c8b056db70
I20260620 23:37:03.596552 10773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61963474ebdd4c8ea0624c31d44bb43b: Bootstrap starting.
I20260620 23:37:03.600668 10773 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61963474ebdd4c8ea0624c31d44bb43b: Neither blocks nor log segments found. Creating new log.
Jun 20 23:37:03 dist-test-slave-tcgx krb5kdc[10693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:37:03.601805 10773 log.cc:824] T 00000000000000000000000000000000 P 61963474ebdd4c8ea0624c31d44bb43b: Log is configured to *not* fsync() on all Append() calls
I20260620 23:37:03.605208 10773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61963474ebdd4c8ea0624c31d44bb43b: No bootstrap required, opened a new log
I20260620 23:37:03.610493 10776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:37:03.588447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38198 (local address 127.0.97.62:37361)
0620 23:37:03.589281 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0620 23:37:03.589293 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:37:03.589347 (+ 54us) server_negotiation.cc:407] Connection header received
0620 23:37:03.591093 (+ 1746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:37:03.591117 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:37:03.591531 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:37:03.591876 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:37:03.593724 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:37:03.595236 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:37:03.597024 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:37:03.597727 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:37:03.597868 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:37:03.602897 (+ 5029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:37:03.602909 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:37:03.602926 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:37:03.603045 (+ 119us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:37:03.606204 (+ 3159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:37:03.606986 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:37:03.606998 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:37:03.607009 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:37:03.607187 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:37:03.607624 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:37:03.607629 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:37:03.607632 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:37:03.608203 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:37:03.608529 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0620 23:37:03.608968 (+ 439us) server_negotiation.cc:299] Negotiation successful
0620 23:37:03.609407 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":393,"threads_started":1}
I20260620 23:37:03.613299 10773 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61963474ebdd4c8ea0624c31d44bb43b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61963474ebdd4c8ea0624c31d44bb43b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37361 } }
I20260620 23:37:03.613758 10773 raft_consensus.cc:374] T 00000000000000000000000000000000 P 61963474ebdd4c8ea0624c31d44bb43b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:37:03.613866 10773 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61963474ebdd4c8ea0624c31d44bb43b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61963474ebdd4c8ea0624c31d44bb43b, State: Initialized, Role: FOLLOWER
I20260620 23:37:03.614420 10773 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61963474ebdd4c8ea0624c31d44bb43b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61963474ebdd4c8ea0624c31d44bb43b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37361 } }
I20260620 23:37:03.614662 10773 raft_consensus.cc:388] T 00000000000000000000000000000000 P 61963474ebdd4c8ea0624c31d44bb43b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:37:03.614799 10773 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61963474ebdd4c8ea0624c31d44bb43b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:37:03.615000 10773 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61963474ebdd4c8ea0624c31d44bb43b [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:37:03.617149 10773 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61963474ebdd4c8ea0624c31d44bb43b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61963474ebdd4c8ea0624c31d44bb43b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37361 } }
I20260620 23:37:03.617718 10773 leader_election.cc:302] T 00000000000000000000000000000000 P 61963474ebdd4c8ea0624c31d44bb43b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61963474ebdd4c8ea0624c31d44bb43b; no voters:
I20260620 23:37:03.618381 10773 leader_election.cc:288] T 00000000000000000000000000000000 P 61963474ebdd4c8ea0624c31d44bb43b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:37:03.618522 10778 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61963474ebdd4c8ea0624c31d44bb43b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:37:03.619648 10778 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61963474ebdd4c8ea0624c31d44bb43b [term 1 LEADER]: Becoming Leader. State: Replica: 61963474ebdd4c8ea0624c31d44bb43b, State: Running, Role: LEADER
I20260620 23:37:03.620306 10778 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61963474ebdd4c8ea0624c31d44bb43b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61963474ebdd4c8ea0624c31d44bb43b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37361 } }
I20260620 23:37:03.620738 10773 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61963474ebdd4c8ea0624c31d44bb43b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:37:03.623279 10780 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61963474ebdd4c8ea0624c31d44bb43b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61963474ebdd4c8ea0624c31d44bb43b. Latest consensus state: current_term: 1 leader_uuid: "61963474ebdd4c8ea0624c31d44bb43b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61963474ebdd4c8ea0624c31d44bb43b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37361 } } }
I20260620 23:37:03.623733 10780 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61963474ebdd4c8ea0624c31d44bb43b [sys.catalog]: This master's current role is: LEADER
I20260620 23:37:03.624364 10779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61963474ebdd4c8ea0624c31d44bb43b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "61963474ebdd4c8ea0624c31d44bb43b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61963474ebdd4c8ea0624c31d44bb43b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37361 } } }
I20260620 23:37:03.624651 10779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61963474ebdd4c8ea0624c31d44bb43b [sys.catalog]: This master's current role is: LEADER
I20260620 23:37:03.626664 10787 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:37:03.631733 10787 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:37:03.642457 10787 catalog_manager.cc:1347] Generated new cluster ID: 16667f88f6ed4df897c941eae32bf2d9
I20260620 23:37:03.642591 10787 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:37:03.654219 10787 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:37:03.655613 10787 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:37:03.663020 10787 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 61963474ebdd4c8ea0624c31d44bb43b: Generated new TSK 0
I20260620 23:37:03.664018 10787 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 23:37:03.726208 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37361
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:37:04.072587 10799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:37:04.072899 10799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:37:04.072994 10799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:37:04.080140 10799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:37:04.080260 10799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:37:04.080404 10799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 23:37:04.090099 10799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37361
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:37:04.091490 10799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:37:04.092738 10799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:37:04.106050 10804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:37:04.106120 10807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:37:04.106148 10805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:37:04.106756 10799 server_base.cc:1034] running on GCE node
I20260620 23:37:04.107491 10799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:37:04.108405 10799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:37:04.109635 10799 hybrid_clock.cc:648] HybridClock initialized: now 1781998624109589 us; error 45 us; skew 500 ppm
I20260620 23:37:04.109973 10799 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:37:04 dist-test-slave-tcgx krb5kdc[10693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998624, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:37:04.113299 10799 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 23:37:04.114751 10799 webserver.cc:460] Webserver started at http://127.0.97.1:46525/ using document root <none> and password file <none>
I20260620 23:37:04.115547 10799 fs_manager.cc:359] Metadata directory not provided
I20260620 23:37:04.115679 10799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:37:04.116019 10799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:37:04.119086 10799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/minicluster-data/ts-0/data/instance:
uuid: "2fec78354b184cd5814a71286a890514"
format_stamp: "Formatted at 2026-06-20 23:37:04 on dist-test-slave-tcgx"
server_key: "1d15f53d278f783c41f5c542e881237a"
server_key_iv: "5aba76669dfbb5d75079ac4f84c52d99"
server_key_version: "encryptionkey@0"
I20260620 23:37:04.119894 10799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/minicluster-data/ts-0/wal/instance:
uuid: "2fec78354b184cd5814a71286a890514"
format_stamp: "Formatted at 2026-06-20 23:37:04 on dist-test-slave-tcgx"
server_key: "1d15f53d278f783c41f5c542e881237a"
server_key_iv: "5aba76669dfbb5d75079ac4f84c52d99"
server_key_version: "encryptionkey@0"
I20260620 23:37:04.125625 10799 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:37:04.129158 10814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:37:04.130589 10799 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:37:04.130775 10799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/minicluster-data/ts-0/wal
uuid: "2fec78354b184cd5814a71286a890514"
format_stamp: "Formatted at 2026-06-20 23:37:04 on dist-test-slave-tcgx"
server_key: "1d15f53d278f783c41f5c542e881237a"
server_key_iv: "5aba76669dfbb5d75079ac4f84c52d99"
server_key_version: "encryptionkey@0"
I20260620 23:37:04.131016 10799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:37:04.146687 10799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:37:04.151284 10799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:37:04.151592 10799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:37:04.152549 10799 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:37:04.153885 10799 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:37:04.153991 10799 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:37:04.154141 10799 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:37:04.154232 10799 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:37:04.186398 10799 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45273
I20260620 23:37:04.186437 10926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45273 every 8 connection(s)
I20260620 23:37:04.187821 10799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
I20260620 23:37:04.193096 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 10799
I20260620 23:37:04.193301 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/minicluster-data/ts-0/wal/instance
I20260620 23:37:04.193919 388 external_mini_cluster.cc:1428] Setting key 373fdf170da552166bdfef68c2ab0950
Jun 20 23:37:04 dist-test-slave-tcgx krb5kdc[10693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998624, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:37:04.209184 10929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:37:04.190365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41209 (local address 127.0.97.62:37361)
0620 23:37:04.190818 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0620 23:37:04.190822 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:37:04.191841 (+ 1019us) server_negotiation.cc:407] Connection header received
0620 23:37:04.193842 (+ 2001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:37:04.193848 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:37:04.193951 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:37:04.194085 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:37:04.195914 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:37:04.196964 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:37:04.198656 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:37:04.199382 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:37:04.199521 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:37:04.203530 (+ 4009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:37:04.203536 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:37:04.203539 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:37:04.203572 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:37:04.206108 (+ 2536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:37:04.206858 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:37:04.206871 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:37:04.206874 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:37:04.207038 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:37:04.207468 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:37:04.207473 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:37:04.207476 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:37:04.207733 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:37:04.207888 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 23:37:04.208593 (+ 705us) server_negotiation.cc:299] Negotiation successful
0620 23:37:04.208826 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":217,"threads_started":1}
I20260620 23:37:04.210224 10930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:37:04.190906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37361 (local address 127.0.97.1:41209)
0620 23:37:04.191674 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0620 23:37:04.191713 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 23:37:04.193571 (+ 1858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:37:04.194345 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:37:04.194354 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:37:04.195060 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:37:04.195730 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:37:04.195753 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:37:04.197177 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:37:04.197181 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:37:04.198493 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:37:04.198506 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:37:04.199620 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:37:04.199625 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:37:04.199839 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:37:04.200584 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:37:04.200607 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:37:04.203288 (+ 2681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:37:04.206261 (+ 2973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:37:04.206271 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:37:04.206289 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:37:04.206711 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:37:04.207146 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:37:04.207150 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:37:04.207154 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:37:04.207351 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:37:04.207864 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:37:04.207872 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:37:04.208226 (+ 354us) client_negotiation.cc:769] Sending connection context
0620 23:37:04.208497 (+ 271us) client_negotiation.cc:240] Negotiation successful
0620 23:37:04.209053 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":263,"threads_started":1}
I20260620 23:37:04.211916 10927 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37361
I20260620 23:37:04.212230 10927 heartbeater.cc:464] Registering TS with master...
I20260620 23:37:04.212980 10927 heartbeater.cc:511] Master 127.0.97.62:37361 requested a full tablet report, sending...
I20260620 23:37:04.215457 10738 ts_manager.cc:194] Registered new tserver with Master: 2fec78354b184cd5814a71286a890514 (127.0.97.1:45273)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 23:37:04.217635 10738 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:41209
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 23:37:04.246824 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37361
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:37:04.588555 10933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:37:04.588826 10933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:37:04.588999 10933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:37:04.596428 10933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:37:04.596547 10933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:37:04.596726 10933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 23:37:04.606483 10933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37361
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:37:04.607893 10933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:37:04.609251 10933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:37:04.622859 10938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:37:04.622771 10941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:37:04.622825 10939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:37:04.623731 10933 server_base.cc:1034] running on GCE node
I20260620 23:37:04.624423 10933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:37:04.625423 10933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:37:04.626649 10933 hybrid_clock.cc:648] HybridClock initialized: now 1781998624626601 us; error 45 us; skew 500 ppm
I20260620 23:37:04.627041 10933 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:37:04 dist-test-slave-tcgx krb5kdc[10693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998624, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:37:04.630091 10933 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 23:37:04.631702 10933 webserver.cc:460] Webserver started at http://127.0.97.2:33773/ using document root <none> and password file <none>
I20260620 23:37:04.632428 10933 fs_manager.cc:359] Metadata directory not provided
I20260620 23:37:04.632571 10933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:37:04.632895 10933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:37:04.635998 10933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/minicluster-data/ts-1/data/instance:
uuid: "2a4e286ff2a54509a31322983749af8b"
format_stamp: "Formatted at 2026-06-20 23:37:04 on dist-test-slave-tcgx"
server_key: "1955efc2fcad299301d9cfa121a8fc89"
server_key_iv: "efc200ea25fd4fd27e70b21c90ce204c"
server_key_version: "encryptionkey@0"
I20260620 23:37:04.636822 10933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/minicluster-data/ts-1/wal/instance:
uuid: "2a4e286ff2a54509a31322983749af8b"
format_stamp: "Formatted at 2026-06-20 23:37:04 on dist-test-slave-tcgx"
server_key: "1955efc2fcad299301d9cfa121a8fc89"
server_key_iv: "efc200ea25fd4fd27e70b21c90ce204c"
server_key_version: "encryptionkey@0"
I20260620 23:37:04.642163 10933 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 23:37:04.645787 10948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:37:04.647269 10933 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:37:04.647440 10933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/minicluster-data/ts-1/wal
uuid: "2a4e286ff2a54509a31322983749af8b"
format_stamp: "Formatted at 2026-06-20 23:37:04 on dist-test-slave-tcgx"
server_key: "1955efc2fcad299301d9cfa121a8fc89"
server_key_iv: "efc200ea25fd4fd27e70b21c90ce204c"
server_key_version: "encryptionkey@0"
I20260620 23:37:04.647640 10933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:37:04.671003 10933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:37:04.675103 10933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:37:04.675400 10933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:37:04.676357 10933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:37:04.677764 10933 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:37:04.677866 10933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:37:04.678016 10933 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:37:04.678086 10933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:37:04.711546 10933 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34063
I20260620 23:37:04.711691 11060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34063 every 8 connection(s)
I20260620 23:37:04.713120 10933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
I20260620 23:37:04.718222 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 10933
I20260620 23:37:04.718415 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/minicluster-data/ts-1/wal/instance
I20260620 23:37:04.718983 388 external_mini_cluster.cc:1428] Setting key 337fc5e8d68703b92bf3e58b0b82d6a3
Jun 20 23:37:04 dist-test-slave-tcgx krb5kdc[10693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998624, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:37:04.736469 11063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:37:04.715902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37035 (local address 127.0.97.62:37361)
0620 23:37:04.716353 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0620 23:37:04.716358 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:37:04.717418 (+ 1060us) server_negotiation.cc:407] Connection header received
0620 23:37:04.719937 (+ 2519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:37:04.719942 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:37:04.720010 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:37:04.720164 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:37:04.722127 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:37:04.722925 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:37:04.725490 (+ 2565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:37:04.725944 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:37:04.726053 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:37:04.730564 (+ 4511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:37:04.730568 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:37:04.730572 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:37:04.730607 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:37:04.733136 (+ 2529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:37:04.733918 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:37:04.733927 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:37:04.733930 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:37:04.734050 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:37:04.734558 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:37:04.734565 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:37:04.734568 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:37:04.734863 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:37:04.735038 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 23:37:04.735870 (+ 832us) server_negotiation.cc:299] Negotiation successful
0620 23:37:04.736089 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":240,"threads_started":1}
I20260620 23:37:04.737464 11064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:37:04.716532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37361 (local address 127.0.97.2:37035)
0620 23:37:04.717234 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0620 23:37:04.717280 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 23:37:04.719618 (+ 2338us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:37:04.720382 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:37:04.720390 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:37:04.721093 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:37:04.721924 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:37:04.721952 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:37:04.723121 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:37:04.723130 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:37:04.725330 (+ 2200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:37:04.725347 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:37:04.726182 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:37:04.726191 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:37:04.726525 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:37:04.727457 (+ 932us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:37:04.727494 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:37:04.730302 (+ 2808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:37:04.733333 (+ 3031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:37:04.733343 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:37:04.733361 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:37:04.733782 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:37:04.734160 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:37:04.734165 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:37:04.734170 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:37:04.734451 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:37:04.735024 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:37:04.735032 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:37:04.735492 (+ 460us) client_negotiation.cc:769] Sending connection context
0620 23:37:04.735792 (+ 300us) client_negotiation.cc:240] Negotiation successful
0620 23:37:04.736346 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":267,"threads_started":1}
I20260620 23:37:04.739061 11061 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37361
I20260620 23:37:04.739451 11061 heartbeater.cc:464] Registering TS with master...
I20260620 23:37:04.740427 11061 heartbeater.cc:511] Master 127.0.97.62:37361 requested a full tablet report, sending...
I20260620 23:37:04.742141 10738 ts_manager.cc:194] Registered new tserver with Master: 2a4e286ff2a54509a31322983749af8b (127.0.97.2:34063)
I20260620 23:37:04.743013 10738 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:37035
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 23:37:04.774094 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37361
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:37:05.113250 11067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:37:05.113570 11067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:37:05.113749 11067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:37:05.121395 11067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:37:05.121562 11067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:37:05.121742 11067 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 23:37:05.132037 11067 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37361
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:37:05.133450 11067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:37:05.135042 11067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:37:05.149329 11072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:37:05.149300 11075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:37:05.149391 11073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:37:05.152735 11067 server_base.cc:1034] running on GCE node
I20260620 23:37:05.153403 11067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:37:05.154516 11067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:37:05.155773 11067 hybrid_clock.cc:648] HybridClock initialized: now 1781998625155732 us; error 44 us; skew 500 ppm
I20260620 23:37:05.156185 11067 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:37:05 dist-test-slave-tcgx krb5kdc[10693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:37:05.159534 11067 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 23:37:05.161155 11067 webserver.cc:460] Webserver started at http://127.0.97.3:37209/ using document root <none> and password file <none>
I20260620 23:37:05.161882 11067 fs_manager.cc:359] Metadata directory not provided
I20260620 23:37:05.161989 11067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:37:05.162243 11067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:37:05.165266 11067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/minicluster-data/ts-2/data/instance:
uuid: "124708edafeb49a595bf7d8ab6b69d77"
format_stamp: "Formatted at 2026-06-20 23:37:05 on dist-test-slave-tcgx"
server_key: "39f7d314c9ced862579ac493b825bf3b"
server_key_iv: "49ff05af20a894e327f1ede7292ef27c"
server_key_version: "encryptionkey@0"
I20260620 23:37:05.165978 11067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/minicluster-data/ts-2/wal/instance:
uuid: "124708edafeb49a595bf7d8ab6b69d77"
format_stamp: "Formatted at 2026-06-20 23:37:05 on dist-test-slave-tcgx"
server_key: "39f7d314c9ced862579ac493b825bf3b"
server_key_iv: "49ff05af20a894e327f1ede7292ef27c"
server_key_version: "encryptionkey@0"
I20260620 23:37:05.171319 11067 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260620 23:37:05.174757 11082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:37:05.176225 11067 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:37:05.176359 11067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/minicluster-data/ts-2/wal
uuid: "124708edafeb49a595bf7d8ab6b69d77"
format_stamp: "Formatted at 2026-06-20 23:37:05 on dist-test-slave-tcgx"
server_key: "39f7d314c9ced862579ac493b825bf3b"
server_key_iv: "49ff05af20a894e327f1ede7292ef27c"
server_key_version: "encryptionkey@0"
I20260620 23:37:05.176530 11067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:37:05.201407 11067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:37:05.206112 11067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:37:05.206372 11067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:37:05.207413 11067 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:37:05.208797 11067 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:37:05.208873 11067 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:37:05.208956 11067 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:37:05.209000 11067 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:37:05.220798 10927 heartbeater.cc:503] Master 127.0.97.62:37361 was elected leader, sending a full tablet report...
I20260620 23:37:05.244498 11067 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41585
I20260620 23:37:05.244547 11194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41585 every 8 connection(s)
I20260620 23:37:05.245878 11067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
I20260620 23:37:05.248518 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 11067
I20260620 23:37:05.248715 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/minicluster-data/ts-2/wal/instance
I20260620 23:37:05.249289 388 external_mini_cluster.cc:1428] Setting key 13ddf93ee3e4f2487db0eeb9920f9511
Jun 20 23:37:05 dist-test-slave-tcgx krb5kdc[10693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:37:05.269661 11197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:37:05.248649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52401 (local address 127.0.97.62:37361)
0620 23:37:05.254783 (+ 6134us) server_negotiation.cc:206] Beginning negotiation
0620 23:37:05.254788 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:37:05.254803 (+ 15us) server_negotiation.cc:407] Connection header received
0620 23:37:05.254854 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:37:05.254858 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:37:05.254916 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:37:05.255074 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:37:05.256747 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:37:05.257525 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:37:05.259255 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:37:05.259721 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:37:05.259827 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:37:05.263412 (+ 3585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:37:05.263417 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:37:05.263420 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:37:05.263454 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:37:05.266439 (+ 2985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:37:05.267316 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:37:05.267328 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:37:05.267331 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:37:05.267466 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:37:05.267900 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:37:05.267911 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:37:05.267913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:37:05.268177 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:37:05.268329 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 23:37:05.269050 (+ 721us) server_negotiation.cc:299] Negotiation successful
0620 23:37:05.269286 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6021,"thread_start_us":5905,"threads_started":1}
I20260620 23:37:05.270650 11198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:37:05.249324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37361 (local address 127.0.97.3:52401)
0620 23:37:05.250508 (+ 1184us) negotiation.cc:107] Waiting for socket to connect
0620 23:37:05.250581 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0620 23:37:05.253934 (+ 3353us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:37:05.255353 (+ 1419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:37:05.255364 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:37:05.255944 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:37:05.256574 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:37:05.256592 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:37:05.257693 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:37:05.257697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:37:05.259092 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:37:05.259105 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:37:05.259924 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:37:05.259928 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:37:05.260129 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:37:05.260850 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:37:05.260872 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:37:05.263184 (+ 2312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:37:05.266620 (+ 3436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:37:05.266630 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:37:05.266656 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:37:05.267189 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:37:05.267576 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:37:05.267582 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:37:05.267586 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:37:05.267780 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:37:05.268310 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:37:05.268318 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:37:05.268678 (+ 360us) client_negotiation.cc:769] Sending connection context
0620 23:37:05.268954 (+ 276us) client_negotiation.cc:240] Negotiation successful
0620 23:37:05.269512 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"thread_start_us":406,"threads_started":1}
I20260620 23:37:05.272198 11195 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37361
I20260620 23:37:05.272562 11195 heartbeater.cc:464] Registering TS with master...
I20260620 23:37:05.273295 11195 heartbeater.cc:511] Master 127.0.97.62:37361 requested a full tablet report, sending...
I20260620 23:37:05.274758 10738 ts_manager.cc:194] Registered new tserver with Master: 124708edafeb49a595bf7d8ab6b69d77 (127.0.97.3:41585)
I20260620 23:37:05.275724 10738 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:52401
I20260620 23:37:05.277603 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:37:05.295073 11197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:37:05.282512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38214 (local address 127.0.97.62:37361)
0620 23:37:05.282762 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 23:37:05.282768 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:37:05.283073 (+ 305us) server_negotiation.cc:407] Connection header received
0620 23:37:05.283231 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:37:05.283235 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:37:05.283298 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:37:05.283382 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:37:05.284684 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:37:05.285430 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:37:05.286633 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:37:05.287175 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:37:05.287285 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:37:05.289610 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:37:05.289615 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:37:05.289619 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:37:05.289653 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:37:05.292495 (+ 2842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:37:05.293183 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:37:05.293187 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:37:05.293189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:37:05.293305 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:37:05.293670 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:37:05.293674 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:37:05.293677 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:37:05.294017 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:37:05.294211 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 23:37:05.294470 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 23:37:05.294653 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260620 23:37:05.299917 10737 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38214:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:37:05.306989 10737 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:37:05.344285 11209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:37:05.320192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34063 (local address 127.0.0.1:37774)
0620 23:37:05.332373 (+ 12181us) negotiation.cc:107] Waiting for socket to connect
0620 23:37:05.332390 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:37:05.332522 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:37:05.333480 (+ 958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:37:05.333485 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:37:05.333512 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:37:05.334130 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:37:05.334148 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:37:05.337244 (+ 3096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:37:05.337252 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:37:05.340375 (+ 3123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:37:05.340396 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:37:05.343121 (+ 2725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:37:05.343127 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:37:05.343308 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:37:05.343338 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 23:37:05.343446 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 23:37:05.343663 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12057,"mutex_wait_us":1,"thread_start_us":11887,"threads_started":1}
I20260620 23:37:05.344494 11211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:37:05.320541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45868 (local address 127.0.97.1:45273)
0620 23:37:05.332058 (+ 11517us) server_negotiation.cc:206] Beginning negotiation
0620 23:37:05.332071 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 23:37:05.332110 (+ 39us) server_negotiation.cc:407] Connection header received
0620 23:37:05.332227 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:37:05.332237 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:37:05.332533 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:37:05.332749 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:37:05.333606 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:37:05.335783 (+ 2177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:37:05.339764 (+ 3981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:37:05.341999 (+ 2235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:37:05.342172 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:37:05.342502 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0620 23:37:05.343823 (+ 1321us) server_negotiation.cc:299] Negotiation successful
0620 23:37:05.344071 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11334,"thread_start_us":11197,"threads_started":1}
I20260620 23:37:05.345402 11208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:37:05.320002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45273 (local address 127.0.0.1:45868)
0620 23:37:05.331470 (+ 11468us) negotiation.cc:107] Waiting for socket to connect
0620 23:37:05.331556 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0620 23:37:05.331839 (+ 283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:37:05.332894 (+ 1055us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:37:05.332904 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:37:05.332941 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:37:05.333335 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:37:05.333348 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:37:05.336005 (+ 2657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:37:05.336013 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:37:05.339484 (+ 3471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:37:05.339506 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:37:05.342692 (+ 3186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:37:05.342702 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:37:05.343039 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:37:05.343120 (+ 81us) client_negotiation.cc:769] Sending connection context
0620 23:37:05.343309 (+ 189us) client_negotiation.cc:240] Negotiation successful
0620 23:37:05.343584 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11013,"mutex_wait_us":2,"thread_start_us":12105,"threads_started":1}
I20260620 23:37:05.345599 11210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:37:05.320331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37774 (local address 127.0.97.2:34063)
0620 23:37:05.331250 (+ 10919us) server_negotiation.cc:206] Beginning negotiation
0620 23:37:05.331266 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 23:37:05.332525 (+ 1259us) server_negotiation.cc:407] Connection header received
0620 23:37:05.332751 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:37:05.332758 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:37:05.332991 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:37:05.333201 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:37:05.334602 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:37:05.337005 (+ 2403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:37:05.340609 (+ 3604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:37:05.342593 (+ 1984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:37:05.342733 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:37:05.342966 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0620 23:37:05.344881 (+ 1915us) server_negotiation.cc:299] Negotiation successful
0620 23:37:05.345155 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10641,"thread_start_us":10918,"threads_started":1}
I20260620 23:37:05.349205 10996 tablet_service.cc:1467] Processing CreateTablet for tablet fe77ea25628040018c494b15c5d46d11 (DEFAULT_TABLE table=test-table [id=616dacbbca9f4c519a87eaedd7d898e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:37:05.350679 10996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe77ea25628040018c494b15c5d46d11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:37:05.354346 10862 tablet_service.cc:1467] Processing CreateTablet for tablet fe77ea25628040018c494b15c5d46d11 (DEFAULT_TABLE table=test-table [id=616dacbbca9f4c519a87eaedd7d898e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:37:05.355279 11208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:37:05.333306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41585 (local address 127.0.0.1:38810)
0620 23:37:05.345921 (+ 12615us) negotiation.cc:107] Waiting for socket to connect
0620 23:37:05.345947 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 23:37:05.346093 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:37:05.347651 (+ 1558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:37:05.347656 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:37:05.347687 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:37:05.347873 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:37:05.347884 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:37:05.350255 (+ 2371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:37:05.350267 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:37:05.352347 (+ 2080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:37:05.352365 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:37:05.354458 (+ 2093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:37:05.354464 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:37:05.354668 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:37:05.354701 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 23:37:05.354794 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 23:37:05.354970 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12462,"thread_start_us":16684,"threads_started":1}
I20260620 23:37:05.355660 11213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:37:05.333400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38810 (local address 127.0.97.3:41585)
0620 23:37:05.346842 (+ 13442us) server_negotiation.cc:206] Beginning negotiation
0620 23:37:05.346856 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 23:37:05.346899 (+ 43us) server_negotiation.cc:407] Connection header received
0620 23:37:05.347058 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:37:05.347071 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:37:05.347406 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:37:05.347665 (+ 259us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:37:05.348042 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:37:05.350110 (+ 2068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:37:05.352539 (+ 2429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:37:05.353994 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:37:05.354122 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:37:05.354352 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0620 23:37:05.355148 (+ 796us) server_negotiation.cc:299] Negotiation successful
0620 23:37:05.355323 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13198,"thread_start_us":13172,"threads_started":1}
I20260620 23:37:05.356074 10862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe77ea25628040018c494b15c5d46d11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:37:05.359556 11130 tablet_service.cc:1467] Processing CreateTablet for tablet fe77ea25628040018c494b15c5d46d11 (DEFAULT_TABLE table=test-table [id=616dacbbca9f4c519a87eaedd7d898e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:37:05.361013 11130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe77ea25628040018c494b15c5d46d11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:37:05.363916 11214 tablet_bootstrap.cc:492] T fe77ea25628040018c494b15c5d46d11 P 2a4e286ff2a54509a31322983749af8b: Bootstrap starting.
I20260620 23:37:05.367149 11214 tablet_bootstrap.cc:654] T fe77ea25628040018c494b15c5d46d11 P 2a4e286ff2a54509a31322983749af8b: Neither blocks nor log segments found. Creating new log.
I20260620 23:37:05.368606 11214 log.cc:824] T fe77ea25628040018c494b15c5d46d11 P 2a4e286ff2a54509a31322983749af8b: Log is configured to *not* fsync() on all Append() calls
I20260620 23:37:05.370560 11216 tablet_bootstrap.cc:492] T fe77ea25628040018c494b15c5d46d11 P 2fec78354b184cd5814a71286a890514: Bootstrap starting.
I20260620 23:37:05.371760 11214 tablet_bootstrap.cc:492] T fe77ea25628040018c494b15c5d46d11 P 2a4e286ff2a54509a31322983749af8b: No bootstrap required, opened a new log
I20260620 23:37:05.372046 11214 ts_tablet_manager.cc:1397] T fe77ea25628040018c494b15c5d46d11 P 2a4e286ff2a54509a31322983749af8b: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260620 23:37:05.374300 11216 tablet_bootstrap.cc:654] T fe77ea25628040018c494b15c5d46d11 P 2fec78354b184cd5814a71286a890514: Neither blocks nor log segments found. Creating new log.
I20260620 23:37:05.375007 11217 tablet_bootstrap.cc:492] T fe77ea25628040018c494b15c5d46d11 P 124708edafeb49a595bf7d8ab6b69d77: Bootstrap starting.
I20260620 23:37:05.375618 11216 log.cc:824] T fe77ea25628040018c494b15c5d46d11 P 2fec78354b184cd5814a71286a890514: Log is configured to *not* fsync() on all Append() calls
I20260620 23:37:05.378297 11217 tablet_bootstrap.cc:654] T fe77ea25628040018c494b15c5d46d11 P 124708edafeb49a595bf7d8ab6b69d77: Neither blocks nor log segments found. Creating new log.
I20260620 23:37:05.378779 11216 tablet_bootstrap.cc:492] T fe77ea25628040018c494b15c5d46d11 P 2fec78354b184cd5814a71286a890514: No bootstrap required, opened a new log
I20260620 23:37:05.379109 11216 ts_tablet_manager.cc:1397] T fe77ea25628040018c494b15c5d46d11 P 2fec78354b184cd5814a71286a890514: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260620 23:37:05.379462 11217 log.cc:824] T fe77ea25628040018c494b15c5d46d11 P 124708edafeb49a595bf7d8ab6b69d77: Log is configured to *not* fsync() on all Append() calls
I20260620 23:37:05.382632 11217 tablet_bootstrap.cc:492] T fe77ea25628040018c494b15c5d46d11 P 124708edafeb49a595bf7d8ab6b69d77: No bootstrap required, opened a new log
I20260620 23:37:05.383008 11217 ts_tablet_manager.cc:1397] T fe77ea25628040018c494b15c5d46d11 P 124708edafeb49a595bf7d8ab6b69d77: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260620 23:37:05.383817 11214 raft_consensus.cc:348] T fe77ea25628040018c494b15c5d46d11 P 2a4e286ff2a54509a31322983749af8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fec78354b184cd5814a71286a890514" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45273 } } peers { permanent_uuid: "2a4e286ff2a54509a31322983749af8b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34063 } } peers { permanent_uuid: "124708edafeb49a595bf7d8ab6b69d77" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41585 } }
I20260620 23:37:05.384295 11214 raft_consensus.cc:374] T fe77ea25628040018c494b15c5d46d11 P 2a4e286ff2a54509a31322983749af8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:37:05.384438 11214 raft_consensus.cc:729] T fe77ea25628040018c494b15c5d46d11 P 2a4e286ff2a54509a31322983749af8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a4e286ff2a54509a31322983749af8b, State: Initialized, Role: FOLLOWER
I20260620 23:37:05.385135 11214 consensus_queue.cc:260] T fe77ea25628040018c494b15c5d46d11 P 2a4e286ff2a54509a31322983749af8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fec78354b184cd5814a71286a890514" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45273 } } peers { permanent_uuid: "2a4e286ff2a54509a31322983749af8b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34063 } } peers { permanent_uuid: "124708edafeb49a595bf7d8ab6b69d77" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41585 } }
I20260620 23:37:05.386348 11061 heartbeater.cc:503] Master 127.0.97.62:37361 was elected leader, sending a full tablet report...
I20260620 23:37:05.387077 11214 ts_tablet_manager.cc:1428] T fe77ea25628040018c494b15c5d46d11 P 2a4e286ff2a54509a31322983749af8b: Time spent starting tablet: real 0.015s user 0.013s sys 0.002s
I20260620 23:37:05.389537 11216 raft_consensus.cc:348] T fe77ea25628040018c494b15c5d46d11 P 2fec78354b184cd5814a71286a890514 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fec78354b184cd5814a71286a890514" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45273 } } peers { permanent_uuid: "2a4e286ff2a54509a31322983749af8b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34063 } } peers { permanent_uuid: "124708edafeb49a595bf7d8ab6b69d77" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41585 } }
I20260620 23:37:05.390457 11216 raft_consensus.cc:374] T fe77ea25628040018c494b15c5d46d11 P 2fec78354b184cd5814a71286a890514 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:37:05.390610 11216 raft_consensus.cc:729] T fe77ea25628040018c494b15c5d46d11 P 2fec78354b184cd5814a71286a890514 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fec78354b184cd5814a71286a890514, State: Initialized, Role: FOLLOWER
I20260620 23:37:05.391345 11216 consensus_queue.cc:260] T fe77ea25628040018c494b15c5d46d11 P 2fec78354b184cd5814a71286a890514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fec78354b184cd5814a71286a890514" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45273 } } peers { permanent_uuid: "2a4e286ff2a54509a31322983749af8b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34063 } } peers { permanent_uuid: "124708edafeb49a595bf7d8ab6b69d77" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41585 } }
I20260620 23:37:05.392167 11217 raft_consensus.cc:348] T fe77ea25628040018c494b15c5d46d11 P 124708edafeb49a595bf7d8ab6b69d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fec78354b184cd5814a71286a890514" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45273 } } peers { permanent_uuid: "2a4e286ff2a54509a31322983749af8b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34063 } } peers { permanent_uuid: "124708edafeb49a595bf7d8ab6b69d77" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41585 } }
I20260620 23:37:05.392644 11217 raft_consensus.cc:374] T fe77ea25628040018c494b15c5d46d11 P 124708edafeb49a595bf7d8ab6b69d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:37:05.392773 11217 raft_consensus.cc:729] T fe77ea25628040018c494b15c5d46d11 P 124708edafeb49a595bf7d8ab6b69d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 124708edafeb49a595bf7d8ab6b69d77, State: Initialized, Role: FOLLOWER
I20260620 23:37:05.393482 11217 consensus_queue.cc:260] T fe77ea25628040018c494b15c5d46d11 P 124708edafeb49a595bf7d8ab6b69d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fec78354b184cd5814a71286a890514" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45273 } } peers { permanent_uuid: "2a4e286ff2a54509a31322983749af8b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34063 } } peers { permanent_uuid: "124708edafeb49a595bf7d8ab6b69d77" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41585 } }
I20260620 23:37:05.394630 11195 heartbeater.cc:503] Master 127.0.97.62:37361 was elected leader, sending a full tablet report...
I20260620 23:37:05.395107 11217 ts_tablet_manager.cc:1428] T fe77ea25628040018c494b15c5d46d11 P 124708edafeb49a595bf7d8ab6b69d77: Time spent starting tablet: real 0.012s user 0.009s sys 0.003s
I20260620 23:37:05.397441 11216 ts_tablet_manager.cc:1428] T fe77ea25628040018c494b15c5d46d11 P 2fec78354b184cd5814a71286a890514: Time spent starting tablet: real 0.018s user 0.016s sys 0.000s
W20260620 23:37:05.440271 10928 tablet.cc:2369] T fe77ea25628040018c494b15c5d46d11 P 2fec78354b184cd5814a71286a890514: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:37:05.465343 11062 tablet.cc:2369] T fe77ea25628040018c494b15c5d46d11 P 2a4e286ff2a54509a31322983749af8b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:37:05.497716 11196 tablet.cc:2369] T fe77ea25628040018c494b15c5d46d11 P 124708edafeb49a595bf7d8ab6b69d77: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:37:05.553303 11221 raft_consensus.cc:482] T fe77ea25628040018c494b15c5d46d11 P 2fec78354b184cd5814a71286a890514 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:37:05.553580 11221 raft_consensus.cc:504] T fe77ea25628040018c494b15c5d46d11 P 2fec78354b184cd5814a71286a890514 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fec78354b184cd5814a71286a890514" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45273 } } peers { permanent_uuid: "2a4e286ff2a54509a31322983749af8b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34063 } } peers { permanent_uuid: "124708edafeb49a595bf7d8ab6b69d77" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41585 } }
I20260620 23:37:05.555327 11221 leader_election.cc:288] T fe77ea25628040018c494b15c5d46d11 P 2fec78354b184cd5814a71286a890514 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a4e286ff2a54509a31322983749af8b (127.0.97.2:34063), 124708edafeb49a595bf7d8ab6b69d77 (127.0.97.3:41585)
I20260620 23:37:05.564733 11210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:37:05.555752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36901 (local address 127.0.97.2:34063)
0620 23:37:05.555960 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 23:37:05.555966 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:37:05.556701 (+ 735us) server_negotiation.cc:407] Connection header received
0620 23:37:05.556895 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:37:05.556900 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:37:05.556978 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:37:05.557079 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:37:05.557920 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:37:05.559069 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:37:05.562234 (+ 3165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:37:05.563509 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:37:05.563652 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:37:05.563760 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 23:37:05.564333 (+ 573us) server_negotiation.cc:299] Negotiation successful
0620 23:37:05.564469 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 23:37:05.564733 11213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:37:05.555839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33135 (local address 127.0.97.3:41585)
0620 23:37:05.556023 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 23:37:05.556031 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:37:05.556700 (+ 669us) server_negotiation.cc:407] Connection header received
0620 23:37:05.556895 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:37:05.556900 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:37:05.556978 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:37:05.557079 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:37:05.557918 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:37:05.559087 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:37:05.562234 (+ 3147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:37:05.563509 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:37:05.563654 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:37:05.563760 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 23:37:05.564333 (+ 573us) server_negotiation.cc:299] Negotiation successful
0620 23:37:05.564469 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 23:37:05.565040 11224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:37:05.555617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34063 (local address 127.0.97.1:36901)
0620 23:37:05.556545 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0620 23:37:05.556574 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 23:37:05.556750 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:37:05.557111 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:37:05.557117 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:37:05.557150 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:37:05.557759 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:37:05.557772 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:37:05.559234 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:37:05.559240 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:37:05.562018 (+ 2778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:37:05.562035 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:37:05.563803 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:37:05.563808 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:37:05.564066 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:37:05.564167 (+ 101us) client_negotiation.cc:769] Sending connection context
0620 23:37:05.564266 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 23:37:05.564501 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":778,"mutex_wait_us":350,"spinlock_wait_cycles":23808,"thread_start_us":600,"threads_started":1}
I20260620 23:37:05.565062 11223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:37:05.555691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41585 (local address 127.0.97.1:33135)
0620 23:37:05.556580 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0620 23:37:05.556594 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:37:05.556750 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:37:05.557111 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:37:05.557117 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:37:05.557148 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:37:05.557674 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:37:05.557695 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:37:05.559235 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:37:05.559240 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:37:05.562018 (+ 2778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:37:05.562035 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:37:05.563791 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:37:05.563797 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:37:05.564070 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:37:05.564172 (+ 102us) client_negotiation.cc:769] Sending connection context
0620 23:37:05.564266 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 23:37:05.564501 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":789,"mutex_wait_us":281,"spinlock_wait_cycles":20480,"thread_start_us":443,"threads_started":1}
I20260620 23:37:05.566249 11016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe77ea25628040018c494b15c5d46d11" candidate_uuid: "2fec78354b184cd5814a71286a890514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a4e286ff2a54509a31322983749af8b" is_pre_election: true
I20260620 23:37:05.566324 11150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe77ea25628040018c494b15c5d46d11" candidate_uuid: "2fec78354b184cd5814a71286a890514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "124708edafeb49a595bf7d8ab6b69d77" is_pre_election: true
I20260620 23:37:05.566862 11016 raft_consensus.cc:2454] T fe77ea25628040018c494b15c5d46d11 P 2a4e286ff2a54509a31322983749af8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fec78354b184cd5814a71286a890514 in term 0.
I20260620 23:37:05.566862 11150 raft_consensus.cc:2454] T fe77ea25628040018c494b15c5d46d11 P 124708edafeb49a595bf7d8ab6b69d77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fec78354b184cd5814a71286a890514 in term 0.
I20260620 23:37:05.567658 10817 leader_election.cc:302] T fe77ea25628040018c494b15c5d46d11 P 2fec78354b184cd5814a71286a890514 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a4e286ff2a54509a31322983749af8b, 2fec78354b184cd5814a71286a890514; no voters:
I20260620 23:37:05.568089 11221 raft_consensus.cc:2781] T fe77ea25628040018c494b15c5d46d11 P 2fec78354b184cd5814a71286a890514 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:37:05.568262 11221 raft_consensus.cc:482] T fe77ea25628040018c494b15c5d46d11 P 2fec78354b184cd5814a71286a890514 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:37:05.568370 11221 raft_consensus.cc:3037] T fe77ea25628040018c494b15c5d46d11 P 2fec78354b184cd5814a71286a890514 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:37:05.570533 11221 raft_consensus.cc:504] T fe77ea25628040018c494b15c5d46d11 P 2fec78354b184cd5814a71286a890514 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fec78354b184cd5814a71286a890514" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45273 } } peers { permanent_uuid: "2a4e286ff2a54509a31322983749af8b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34063 } } peers { permanent_uuid: "124708edafeb49a595bf7d8ab6b69d77" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41585 } }
I20260620 23:37:05.571221 11221 leader_election.cc:288] T fe77ea25628040018c494b15c5d46d11 P 2fec78354b184cd5814a71286a890514 [CANDIDATE]: Term 1 election: Requested vote from peers 2a4e286ff2a54509a31322983749af8b (127.0.97.2:34063), 124708edafeb49a595bf7d8ab6b69d77 (127.0.97.3:41585)
I20260620 23:37:05.571717 11150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe77ea25628040018c494b15c5d46d11" candidate_uuid: "2fec78354b184cd5814a71286a890514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "124708edafeb49a595bf7d8ab6b69d77"
I20260620 23:37:05.571717 11016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe77ea25628040018c494b15c5d46d11" candidate_uuid: "2fec78354b184cd5814a71286a890514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a4e286ff2a54509a31322983749af8b"
I20260620 23:37:05.571928 11016 raft_consensus.cc:3037] T fe77ea25628040018c494b15c5d46d11 P 2a4e286ff2a54509a31322983749af8b [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:37:05.571929 11150 raft_consensus.cc:3037] T fe77ea25628040018c494b15c5d46d11 P 124708edafeb49a595bf7d8ab6b69d77 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:37:05.575116 11016 raft_consensus.cc:2454] T fe77ea25628040018c494b15c5d46d11 P 2a4e286ff2a54509a31322983749af8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fec78354b184cd5814a71286a890514 in term 1.
I20260620 23:37:05.575119 11150 raft_consensus.cc:2454] T fe77ea25628040018c494b15c5d46d11 P 124708edafeb49a595bf7d8ab6b69d77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fec78354b184cd5814a71286a890514 in term 1.
I20260620 23:37:05.575532 10817 leader_election.cc:302] T fe77ea25628040018c494b15c5d46d11 P 2fec78354b184cd5814a71286a890514 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a4e286ff2a54509a31322983749af8b, 2fec78354b184cd5814a71286a890514; no voters:
I20260620 23:37:05.575850 11221 raft_consensus.cc:2781] T fe77ea25628040018c494b15c5d46d11 P 2fec78354b184cd5814a71286a890514 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:37:05.576287 11221 raft_consensus.cc:686] T fe77ea25628040018c494b15c5d46d11 P 2fec78354b184cd5814a71286a890514 [term 1 LEADER]: Becoming Leader. State: Replica: 2fec78354b184cd5814a71286a890514, State: Running, Role: LEADER
I20260620 23:37:05.576750 11221 consensus_queue.cc:237] T fe77ea25628040018c494b15c5d46d11 P 2fec78354b184cd5814a71286a890514 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fec78354b184cd5814a71286a890514" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45273 } } peers { permanent_uuid: "2a4e286ff2a54509a31322983749af8b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34063 } } peers { permanent_uuid: "124708edafeb49a595bf7d8ab6b69d77" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41585 } }
I20260620 23:37:05.581569 10738 catalog_manager.cc:5495] T fe77ea25628040018c494b15c5d46d11 P 2fec78354b184cd5814a71286a890514 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2fec78354b184cd5814a71286a890514 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "2fec78354b184cd5814a71286a890514" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fec78354b184cd5814a71286a890514" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45273 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a4e286ff2a54509a31322983749af8b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "124708edafeb49a595bf7d8ab6b69d77" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41585 } health_report { overall_health: UNKNOWN } } }
I20260620 23:37:05.599431 11211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:37:05.594251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38790 (local address 127.0.97.1:45273)
0620 23:37:05.594441 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 23:37:05.594447 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:37:05.594466 (+ 19us) server_negotiation.cc:407] Connection header received
0620 23:37:05.594686 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:37:05.594690 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:37:05.594788 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:37:05.594915 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:37:05.595328 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:37:05.596028 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:37:05.597305 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:37:05.597723 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:37:05.597821 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:37:05.598298 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:37:05.598804 (+ 506us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:37:05.598910 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 23:37:05.599068 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 23:37:05.599136 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 23:37:05.607788 11016 raft_consensus.cc:1261] T fe77ea25628040018c494b15c5d46d11 P 2a4e286ff2a54509a31322983749af8b [term 1 FOLLOWER]: Refusing update from remote peer 2fec78354b184cd5814a71286a890514: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:37:05.607788 11150 raft_consensus.cc:1261] T fe77ea25628040018c494b15c5d46d11 P 124708edafeb49a595bf7d8ab6b69d77 [term 1 FOLLOWER]: Refusing update from remote peer 2fec78354b184cd5814a71286a890514: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:37:05.609052 11225 consensus_queue.cc:1035] T fe77ea25628040018c494b15c5d46d11 P 2fec78354b184cd5814a71286a890514 [LEADER]: Connected to new peer: Peer: permanent_uuid: "124708edafeb49a595bf7d8ab6b69d77" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:37:05.609463 11221 consensus_queue.cc:1035] T fe77ea25628040018c494b15c5d46d11 P 2fec78354b184cd5814a71286a890514 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a4e286ff2a54509a31322983749af8b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:37:05.620033 11228 mvcc.cc:205] Tried to move back new op lower bound from 7299066370478108672 to 7299066370366234624. Current Snapshot: MvccSnapshot[applied={T|T < 7299066370478108672}]
I20260620 23:37:05.636673 10737 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:37:05.636978 10737 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:37:05.640707 10737 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 61963474ebdd4c8ea0624c31d44bb43b: Sending DeleteTablet for 3 replicas of tablet fe77ea25628040018c494b15c5d46d11
I20260620 23:37:05.641939 10996 tablet_service.cc:1514] Processing DeleteTablet for tablet fe77ea25628040018c494b15c5d46d11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:37:05 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37774
I20260620 23:37:05.641958 11130 tablet_service.cc:1514] Processing DeleteTablet for tablet fe77ea25628040018c494b15c5d46d11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:37:05 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38810
I20260620 23:37:05.642921 10862 tablet_service.cc:1514] Processing DeleteTablet for tablet fe77ea25628040018c494b15c5d46d11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:37:05 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45868
I20260620 23:37:05.643433 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 10799
I20260620 23:37:05.666047 11239 tablet_replica.cc:330] T fe77ea25628040018c494b15c5d46d11 P 2fec78354b184cd5814a71286a890514: stopping tablet replica
I20260620 23:37:05.667153 11239 raft_consensus.cc:2229] T fe77ea25628040018c494b15c5d46d11 P 2fec78354b184cd5814a71286a890514 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:37:05.668114 11239 raft_consensus.cc:2258] T fe77ea25628040018c494b15c5d46d11 P 2fec78354b184cd5814a71286a890514 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:37:05.669534 11237 tablet_replica.cc:330] T fe77ea25628040018c494b15c5d46d11 P 2a4e286ff2a54509a31322983749af8b: stopping tablet replica
I20260620 23:37:05.669979 11237 raft_consensus.cc:2229] T fe77ea25628040018c494b15c5d46d11 P 2a4e286ff2a54509a31322983749af8b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:37:05.670481 11237 raft_consensus.cc:2258] T fe77ea25628040018c494b15c5d46d11 P 2a4e286ff2a54509a31322983749af8b [term 1 FOLLOWER]: Raft consensus is shut down!
Jun 20 23:37:05 dist-test-slave-tcgx krb5kdc[10693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 23:37:05.671116 11238 tablet_replica.cc:330] T fe77ea25628040018c494b15c5d46d11 P 124708edafeb49a595bf7d8ab6b69d77: stopping tablet replica
I20260620 23:37:05.671525 11238 raft_consensus.cc:2229] T fe77ea25628040018c494b15c5d46d11 P 124708edafeb49a595bf7d8ab6b69d77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:37:05.671839 11239 ts_tablet_manager.cc:1905] T fe77ea25628040018c494b15c5d46d11 P 2fec78354b184cd5814a71286a890514: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:37:05.671984 11238 raft_consensus.cc:2258] T fe77ea25628040018c494b15c5d46d11 P 124708edafeb49a595bf7d8ab6b69d77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:37:05.674753 11238 ts_tablet_manager.cc:1905] T fe77ea25628040018c494b15c5d46d11 P 124708edafeb49a595bf7d8ab6b69d77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:37:05.674966 11237 ts_tablet_manager.cc:1905] T fe77ea25628040018c494b15c5d46d11 P 2a4e286ff2a54509a31322983749af8b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:37:05.682054 11237 ts_tablet_manager.cc:1918] T fe77ea25628040018c494b15c5d46d11 P 2a4e286ff2a54509a31322983749af8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:37:05.682301 11237 log.cc:1196] T fe77ea25628040018c494b15c5d46d11 P 2a4e286ff2a54509a31322983749af8b: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/minicluster-data/ts-1/wal/wals/fe77ea25628040018c494b15c5d46d11
I20260620 23:37:05.682322 11238 ts_tablet_manager.cc:1918] T fe77ea25628040018c494b15c5d46d11 P 124708edafeb49a595bf7d8ab6b69d77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:37:05.682888 11238 log.cc:1196] T fe77ea25628040018c494b15c5d46d11 P 124708edafeb49a595bf7d8ab6b69d77: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/minicluster-data/ts-2/wal/wals/fe77ea25628040018c494b15c5d46d11
I20260620 23:37:05.682996 11237 ts_tablet_manager.cc:1939] T fe77ea25628040018c494b15c5d46d11 P 2a4e286ff2a54509a31322983749af8b: Deleting consensus metadata
I20260620 23:37:05.683606 11238 ts_tablet_manager.cc:1939] T fe77ea25628040018c494b15c5d46d11 P 124708edafeb49a595bf7d8ab6b69d77: Deleting consensus metadata
I20260620 23:37:05.685005 10724 catalog_manager.cc:4841] TS 2a4e286ff2a54509a31322983749af8b (127.0.97.2:34063): tablet fe77ea25628040018c494b15c5d46d11 (table test-table [id=616dacbbca9f4c519a87eaedd7d898e4]) successfully deleted
I20260620 23:37:05.685346 10722 catalog_manager.cc:4841] TS 124708edafeb49a595bf7d8ab6b69d77 (127.0.97.3:41585): tablet fe77ea25628040018c494b15c5d46d11 (table test-table [id=616dacbbca9f4c519a87eaedd7d898e4]) successfully deleted
I20260620 23:37:05.682647 11239 ts_tablet_manager.cc:1918] T fe77ea25628040018c494b15c5d46d11 P 2fec78354b184cd5814a71286a890514: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:37:05.685770 11239 log.cc:1196] T fe77ea25628040018c494b15c5d46d11 P 2fec78354b184cd5814a71286a890514: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998525501163-388-0/minicluster-data/ts-0/wal/wals/fe77ea25628040018c494b15c5d46d11
I20260620 23:37:05.686370 11239 ts_tablet_manager.cc:1939] T fe77ea25628040018c494b15c5d46d11 P 2fec78354b184cd5814a71286a890514: Deleting consensus metadata
I20260620 23:37:05.687999 10722 catalog_manager.cc:4841] TS 2fec78354b184cd5814a71286a890514 (127.0.97.1:45273): tablet fe77ea25628040018c494b15c5d46d11 (table test-table [id=616dacbbca9f4c519a87eaedd7d898e4]) successfully deleted
I20260620 23:37:05.688845 11211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:37:05.645182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38802 (local address 127.0.97.1:45273)
0620 23:37:05.645415 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 23:37:05.645423 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:37:05.656566 (+ 11143us) server_negotiation.cc:407] Connection header received
0620 23:37:05.657186 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:37:05.657195 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:37:05.657323 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:37:05.657425 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:37:05.659697 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:37:05.661389 (+ 1692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:37:05.663966 (+ 2577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:37:05.664971 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:37:05.665118 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:37:05.680410 (+ 15292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:37:05.680427 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:37:05.680449 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:37:05.680494 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:37:05.684672 (+ 4178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:37:05.685629 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:37:05.685641 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:37:05.685647 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:37:05.685852 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:37:05.686397 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:37:05.686403 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:37:05.686405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:37:05.686901 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:37:05.687230 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0620 23:37:05.688410 (+ 1180us) server_negotiation.cc:299] Negotiation successful
0620 23:37:05.688581 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 23:37:05.689512 10922 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:37:05.789144 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 10799
I20260620 23:37:05.807319 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 10933
Jun 20 23:37:05 dist-test-slave-tcgx krb5kdc[10693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 23:37:05.822086 11210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:37:05.808151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55512 (local address 127.0.97.2:34063)
0620 23:37:05.808345 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 23:37:05.808352 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:37:05.808372 (+ 20us) server_negotiation.cc:407] Connection header received
0620 23:37:05.808560 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:37:05.808566 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:37:05.808644 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:37:05.808751 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:37:05.809917 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:37:05.810681 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:37:05.811986 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:37:05.812402 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:37:05.812534 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:37:05.816543 (+ 4009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:37:05.816557 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:37:05.816573 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:37:05.816607 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:37:05.819274 (+ 2667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:37:05.819915 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:37:05.819921 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:37:05.819924 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:37:05.820065 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:37:05.820434 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:37:05.820441 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:37:05.820444 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:37:05.820863 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:37:05.821177 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0620 23:37:05.821485 (+ 308us) server_negotiation.cc:299] Negotiation successful
0620 23:37:05.821700 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 23:37:05.822779 11056 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:37:05.913911 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 10933
I20260620 23:37:05.931056 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 11067
Jun 20 23:37:05 dist-test-slave-tcgx krb5kdc[10693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 23:37:05.946851 11213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:37:05.931813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40704 (local address 127.0.97.3:41585)
0620 23:37:05.931967 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 23:37:05.931972 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:37:05.931990 (+ 18us) server_negotiation.cc:407] Connection header received
0620 23:37:05.932155 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:37:05.932159 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:37:05.932226 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:37:05.932318 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:37:05.933504 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:37:05.934195 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:37:05.935564 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:37:05.935974 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:37:05.936070 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:37:05.940617 (+ 4547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:37:05.940648 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:37:05.940665 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:37:05.940711 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:37:05.943831 (+ 3120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:37:05.944574 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:37:05.944583 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:37:05.944586 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:37:05.944735 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:37:05.945267 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:37:05.945273 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:37:05.945282 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:37:05.945634 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:37:05.945970 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0620 23:37:05.946279 (+ 309us) server_negotiation.cc:299] Negotiation successful
0620 23:37:05.946485 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 23:37:05.947616 11190 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:37:06.040758 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 11067
I20260620 23:37:06.059612 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 10706
I20260620 23:37:06.060818 10768 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:37:06.170565 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 10706
2026-06-20T23:37:06Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4368 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:37:06 dist-test-slave-tcgx krb5kdc[11247](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:37:06 dist-test-slave-tcgx krb5kdc[11247](info): set up 2 sockets
Jun 20 23:37:06 dist-test-slave-tcgx krb5kdc[11247](info): commencing operation
krb5kdc: starting...
W20260620 23:37:07.275969 388 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/krb5kdc/test-user.keytab.
Jun 20 23:37:07 dist-test-slave-tcgx krb5kdc[11247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:37:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:37:07Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260620 23:37:07.618753 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskN6eSV7/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/postgres -l logfile start
2026-06-20 23:37:10.139 UTC [11277] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 23:37:10.139 UTC [11277] LOG: listening on IPv4 address "127.0.97.20", port 40355
2026-06-20 23:37:10.143 UTC [11277] LOG: listening on Unix socket "/tmp/.s.PGSQL.40355"
2026-06-20 23:37:10.166 UTC [11280] LOG: database system was shut down at 2026-06-20 23:37:08 UTC
2026-06-20 23:37:10.169 UTC [11277] LOG: database system is ready to accept connections
I20260620 23:37:11.151747 388 mini_postgres.cc:96] Postgres bound to 40355
2026-06-20 23:37:11.170 UTC [11288] FATAL: database "slave" does not exist
127.0.97.20:40355 - accepting connections
I20260620 23:37:11.170992 388 mini_ranger.cc:161] Starting Ranger...
I20260620 23:37:11.198220 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 23:37:11.690146 388 mini_ranger.cc:87] Created ranger Postgres database
I20260620 23:37:11.690330 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/ranger-admin
2026-06-20 23:37:11,989 [I] DB FLAVOR :POSTGRES
2026-06-20 23:37:11,990 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 23:37:11,990 [I] Checking connection..
2026-06-20 23:37:11,990 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN6eSV7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN6eSV7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:40355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 23:37:12,251 [I] Checking connection passed.
2026-06-20 23:37:12,251 [I] --------- Verifying version history table ---------
2026-06-20 23:37:12,251 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN6eSV7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN6eSV7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:40355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 23:37:12,519 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 23:37:12,519 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN6eSV7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN6eSV7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:40355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 23:37:12,809 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 23:37:12,810 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 23:37:12,810 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN6eSV7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN6eSV7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:40355/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskN6eSV7/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 23:37:13.074 UTC [11352] WARNING: there is no transaction in progress
2026-06-20 23:37:13,081 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 23:37:13,081 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN6eSV7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN6eSV7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:40355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 23:37:13,338 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 23:37:13,339 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 23:37:13,339 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN6eSV7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN6eSV7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:40355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 23:37:13,597 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN6eSV7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN6eSV7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:40355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 23:37:13,849 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN6eSV7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN6eSV7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:40355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-tcgx.c.gcp-upstream.internal','N') ;"
2026-06-20 23:37:14,122 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN6eSV7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN6eSV7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:40355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 23:37:14,387 [I] Table x_portal_user does not exist in database ranger
2026-06-20 23:37:14,387 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN6eSV7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN6eSV7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:40355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 23:37:14,653 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 23:37:14,653 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN6eSV7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN6eSV7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:40355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 23:37:14,902 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN6eSV7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN6eSV7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:40355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 23:37:15,143 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 23:37:15,143 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN6eSV7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN6eSV7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:40355/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskN6eSV7/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 23:37:16.137 UTC [11460] WARNING: there is no transaction in progress
2026-06-20 23:37:16.154 UTC [11460] WARNING: there is no transaction in progress
2026-06-20 23:37:16.171 UTC [11460] WARNING: there is no transaction in progress
2026-06-20 23:37:16.187 UTC [11460] WARNING: there is no transaction in progress
2026-06-20 23:37:16.204 UTC [11460] WARNING: there is no transaction in progress
2026-06-20 23:37:16.223 UTC [11460] WARNING: there is no transaction in progress
2026-06-20 23:37:16.288 UTC [11460] WARNING: there is no transaction in progress
2026-06-20 23:37:16.295 UTC [11460] WARNING: there is no transaction in progress
2026-06-20 23:37:16.310 UTC [11460] WARNING: there is no transaction in progress
2026-06-20 23:37:16.320 UTC [11460] WARNING: there is no transaction in progress
2026-06-20 23:37:16.330 UTC [11460] WARNING: there is no transaction in progress
2026-06-20 23:37:16.345 UTC [11460] WARNING: there is no transaction in progress
2026-06-20 23:37:16.355 UTC [11460] WARNING: there is no transaction in progress
2026-06-20 23:37:16.361 UTC [11460] WARNING: there is no transaction in progress
2026-06-20 23:37:16.368 UTC [11460] WARNING: there is no transaction in progress
2026-06-20 23:37:16.374 UTC [11460] WARNING: there is no transaction in progress
2026-06-20 23:37:17,058 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 23:37:17,058 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN6eSV7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN6eSV7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:40355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 23:37:17,284 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 23:37:17,284 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN6eSV7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN6eSV7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:40355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 23:37:17,554 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 23:37:17,554 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN6eSV7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN6eSV7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:40355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 23:37:17,809 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 23:37:17,809 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN6eSV7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN6eSV7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:40355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 23:37:18,075 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN6eSV7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN6eSV7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:40355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 23:37:18,343 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN6eSV7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN6eSV7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:40355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-tcgx.c.gcp-upstream.internal';"
2026-06-20 23:37:18,607 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 23:37:18,607 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN6eSV7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN6eSV7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:40355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 23:37:18,874 [I] DB_PATCHES have already been applied
I20260620 23:37:18.880293 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskN6eSV7/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskN6eSV7/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskN6eSV7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN6eSV7/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 23:37:18.880407 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260620 23:37:18.895972 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:40371
Jun 20, 2026 11:37:19 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 11:37:19 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 11:37:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 11:37:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 11:37:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 11:37:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 11:37:19 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 23:37:20.036629 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.866s user 0.011s sys 0.031s
Jun 20, 2026 11:37:39 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 11:37:39 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Jun 20, 2026 11:37:39 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 11:37:39 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 11:37:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 11:37:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 11:37:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 11:37:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 11:37:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 11:37:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 11:37:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 11:37:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 11:37:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 11:37:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 11:37:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 11:37:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 11:37:39 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 11:37:40 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 23:37:41.045590 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 23:37:41 dist-test-slave-tcgx krb5kdc[11247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998661, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 23:37:41.220574 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 23:37:41.242043 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 23:37:42.127249 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 23:37:42.127386 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/ranger-kms
I20260620 23:37:42.127535 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:40355
2026-06-20 23:37:42,553 [I] DB FLAVOR :POSTGRES
2026-06-20 23:37:42,554 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 23:37:42,554 [I] Checking connection
2026-06-20 23:37:42,554 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN6eSV7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:40355/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 23:37:42,804 [I] connection success
2026-06-20 23:37:42,805 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 23:37:42,805 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 23:37:42,805 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN6eSV7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:40355/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 23:37:43,052 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 23:37:43,052 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 23:37:43,053 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 23:37:43,053 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN6eSV7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:40355/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 23:37:43,309 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 23:37:43.798099 388 mini_ranger_kms.cc:310] Created kms service
I20260620 23:37:43.964064 388 mini_ranger_kms.cc:326] Created kudu user
I20260620 23:37:44.031344 388 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 23:37:44 dist-test-slave-tcgx krb5kdc[11247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998661, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 23:37:44.441957 388 mini_ranger_kms.cc:384] Added ranger policy
I20260620 23:37:44.442270 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskN6eSV7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN6eSV7/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskN6eSV7/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 23:37:44.442337 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260620 23:37:44.471091 388 mini_ranger_kms.cc:276] Ranger KMS PID: 11640
I20260620 23:37:44.471362 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:53589
Jun 20, 2026 11:37:45 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 11:37:45 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 11:37:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 11:37:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 11:37:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 11:37:46 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 23:37:46.879189 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.659s user 0.141s sys 0.308s
I20260620 23:37:46.879446 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 23:37:46.879663 388 mini_ranger_kms.cc:401] 127.0.97.20:53589/kms/v1/keys
Jun 20 23:37:47 dist-test-slave-tcgx krb5kdc[11247](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781998667, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 23:37:47 dist-test-slave-tcgx krb5kdc[11247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998667, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 20 23:37:47 dist-test-slave-tcgx krb5kdc[11247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998667, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 20 23:37:50 dist-test-slave-tcgx krb5kdc[11247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 23:37:50.191648 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46603
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46603
--ranger_config_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:53589/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:37:50.545336 11691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:37:50.545713 11691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:37:50.545912 11691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:37:50.553633 11691 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:37:50.553757 11691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:37:50.553802 11691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:37:50.553857 11691 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:37:50.553905 11691 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:37:50.564280 11691 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:53589/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46603
--ranger_config_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:37:50.565681 11691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:37:50.567128 11691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:37:50.580828 11697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:37:50.580979 11699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:37:50.580936 11696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:37:50.581487 11691 server_base.cc:1034] running on GCE node
I20260620 23:37:50.582314 11691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:37:50.583894 11691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:37:50.585106 11691 hybrid_clock.cc:648] HybridClock initialized: now 1781998670585069 us; error 38 us; skew 500 ppm
I20260620 23:37:50.585513 11691 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:37:50 dist-test-slave-tcgx krb5kdc[11247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998670, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:37:50.593551 11691 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 23:37:50.595132 11691 webserver.cc:460] Webserver started at http://127.0.97.62:40487/ using document root <none> and password file <none>
I20260620 23:37:50.595810 11691 fs_manager.cc:359] Metadata directory not provided
I20260620 23:37:50.595907 11691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:37:50.596166 11691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 23:37:50 dist-test-slave-tcgx krb5kdc[11247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998670, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 23:37:50.656840 11691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/master-0/data/instance:
uuid: "88e3f05172b14287ae49b3a58f67f829"
format_stamp: "Formatted at 2026-06-20 23:37:50 on dist-test-slave-tcgx"
server_key: "31324c807486ad9c0e40cc12e8ba4215"
server_key_iv: "89697f1c8eb774257f5541d09023a8e9"
server_key_version: "kuduclusterkey@0"
I20260620 23:37:50.658335 11691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/master-0/wal/instance:
uuid: "88e3f05172b14287ae49b3a58f67f829"
format_stamp: "Formatted at 2026-06-20 23:37:50 on dist-test-slave-tcgx"
server_key: "31324c807486ad9c0e40cc12e8ba4215"
server_key_iv: "89697f1c8eb774257f5541d09023a8e9"
server_key_version: "kuduclusterkey@0"
I20260620 23:37:50.664597 11691 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260620 23:37:50.685814 11706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:37:50.687754 11691 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260620 23:37:50.687964 11691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/master-0/wal
uuid: "88e3f05172b14287ae49b3a58f67f829"
format_stamp: "Formatted at 2026-06-20 23:37:50 on dist-test-slave-tcgx"
server_key: "31324c807486ad9c0e40cc12e8ba4215"
server_key_iv: "89697f1c8eb774257f5541d09023a8e9"
server_key_version: "kuduclusterkey@0"
I20260620 23:37:50.688232 11691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:37:50.710359 11691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:37:50.715648 11691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:37:50.715972 11691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:37:50.736652 11691 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46603
I20260620 23:37:50.736653 11757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46603 every 8 connection(s)
I20260620 23:37:50.738497 11691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:37:50.739849 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 11691
I20260620 23:37:50.740065 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/master-0/wal/instance
I20260620 23:37:50.740749 388 external_mini_cluster.cc:1428] Setting key 1b1866aa5eac87b6246ae638c290683f
I20260620 23:37:50.744165 11758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 23:37:50 dist-test-slave-tcgx krb5kdc[11247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:37:50.757431 11758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88e3f05172b14287ae49b3a58f67f829: Bootstrap starting.
I20260620 23:37:50.760646 11758 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88e3f05172b14287ae49b3a58f67f829: Neither blocks nor log segments found. Creating new log.
I20260620 23:37:50.761667 11758 log.cc:824] T 00000000000000000000000000000000 P 88e3f05172b14287ae49b3a58f67f829: Log is configured to *not* fsync() on all Append() calls
I20260620 23:37:50.764034 11761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:37:50.743335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58168 (local address 127.0.97.62:46603)
0620 23:37:50.744103 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0620 23:37:50.744115 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:37:50.744163 (+ 48us) server_negotiation.cc:407] Connection header received
0620 23:37:50.745759 (+ 1596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:37:50.745779 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:37:50.746182 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:37:50.746511 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:37:50.748105 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:37:50.749524 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:37:50.751312 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:37:50.751968 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:37:50.752095 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:37:50.756526 (+ 4431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:37:50.756534 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:37:50.756550 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:37:50.756587 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:37:50.759685 (+ 3098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:37:50.760380 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:37:50.760386 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:37:50.760392 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:37:50.760546 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:37:50.761014 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:37:50.761018 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:37:50.761020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:37:50.761525 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:37:50.761836 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0620 23:37:50.762278 (+ 442us) server_negotiation.cc:299] Negotiation successful
0620 23:37:50.762726 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":352,"threads_started":1}
I20260620 23:37:50.765089 11758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88e3f05172b14287ae49b3a58f67f829: No bootstrap required, opened a new log
I20260620 23:37:50.773669 11758 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88e3f05172b14287ae49b3a58f67f829 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e3f05172b14287ae49b3a58f67f829" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46603 } }
I20260620 23:37:50.774133 11758 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88e3f05172b14287ae49b3a58f67f829 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:37:50.774237 11758 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88e3f05172b14287ae49b3a58f67f829 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88e3f05172b14287ae49b3a58f67f829, State: Initialized, Role: FOLLOWER
I20260620 23:37:50.774816 11758 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88e3f05172b14287ae49b3a58f67f829 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e3f05172b14287ae49b3a58f67f829" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46603 } }
I20260620 23:37:50.775128 11758 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88e3f05172b14287ae49b3a58f67f829 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:37:50.775272 11758 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88e3f05172b14287ae49b3a58f67f829 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:37:50.775449 11758 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88e3f05172b14287ae49b3a58f67f829 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:37:50.777915 11758 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88e3f05172b14287ae49b3a58f67f829 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e3f05172b14287ae49b3a58f67f829" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46603 } }
I20260620 23:37:50.778445 11758 leader_election.cc:302] T 00000000000000000000000000000000 P 88e3f05172b14287ae49b3a58f67f829 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88e3f05172b14287ae49b3a58f67f829; no voters:
I20260620 23:37:50.779075 11758 leader_election.cc:288] T 00000000000000000000000000000000 P 88e3f05172b14287ae49b3a58f67f829 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:37:50.779215 11763 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88e3f05172b14287ae49b3a58f67f829 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:37:50.779875 11763 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88e3f05172b14287ae49b3a58f67f829 [term 1 LEADER]: Becoming Leader. State: Replica: 88e3f05172b14287ae49b3a58f67f829, State: Running, Role: LEADER
I20260620 23:37:50.780462 11763 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88e3f05172b14287ae49b3a58f67f829 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e3f05172b14287ae49b3a58f67f829" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46603 } }
I20260620 23:37:50.781210 11758 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88e3f05172b14287ae49b3a58f67f829 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:37:50.783926 11765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88e3f05172b14287ae49b3a58f67f829 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88e3f05172b14287ae49b3a58f67f829. Latest consensus state: current_term: 1 leader_uuid: "88e3f05172b14287ae49b3a58f67f829" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e3f05172b14287ae49b3a58f67f829" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46603 } } }
I20260620 23:37:50.784301 11765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88e3f05172b14287ae49b3a58f67f829 [sys.catalog]: This master's current role is: LEADER
I20260620 23:37:50.784552 11764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88e3f05172b14287ae49b3a58f67f829 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88e3f05172b14287ae49b3a58f67f829" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88e3f05172b14287ae49b3a58f67f829" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46603 } } }
I20260620 23:37:50.784838 11764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88e3f05172b14287ae49b3a58f67f829 [sys.catalog]: This master's current role is: LEADER
I20260620 23:37:50.786885 11771 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:37:50.792598 11771 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:37:50.793428 11758 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 23:37:50.811899 11771 catalog_manager.cc:1347] Generated new cluster ID: 63f6ad31a9ff4f8d99986b1c23152b93
I20260620 23:37:50.812073 11771 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:37:50.835686 11771 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:37:50.838346 11771 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:37:50.851835 11771 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88e3f05172b14287ae49b3a58f67f829: Generated new TSK 0
I20260620 23:37:50.853034 11771 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 23:37:51.661340 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:53589/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46603
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:37:51 dist-test-slave-tcgx krb5kdc[11247](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781998671, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 23:37:52.119066 11803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:37:52.119535 11803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:37:52.119778 11803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:37:52.132120 11803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:37:52.132293 11803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:37:52.132496 11803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 23:37:52.149082 11803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:53589/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46603
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:37:52.150985 11803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:37:52.152817 11803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:37:52.172858 11813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:37:52.173740 11814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:37:52.179399 11816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:37:52.180027 11803 server_base.cc:1034] running on GCE node
I20260620 23:37:52.180819 11803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:37:52.181965 11803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:37:52.183200 11803 hybrid_clock.cc:648] HybridClock initialized: now 1781998672183141 us; error 55 us; skew 500 ppm
I20260620 23:37:52.183554 11803 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:37:52 dist-test-slave-tcgx krb5kdc[11247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:37:52.187325 11803 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 23:37:52.189158 11803 webserver.cc:460] Webserver started at http://127.0.97.1:36503/ using document root <none> and password file <none>
I20260620 23:37:52.189931 11803 fs_manager.cc:359] Metadata directory not provided
I20260620 23:37:52.190081 11803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:37:52.190426 11803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 23:37:52 dist-test-slave-tcgx krb5kdc[11247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 23:37:52.209692 11803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/ts-0/data/instance:
uuid: "031d7a7279e54c6180ab5144f10a8c09"
format_stamp: "Formatted at 2026-06-20 23:37:52 on dist-test-slave-tcgx"
server_key: "425441eff8a3f4948e1540a375b69769"
server_key_iv: "ce240b27fd1380369f35bac76e766249"
server_key_version: "kuduclusterkey@0"
I20260620 23:37:52.211014 11803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/ts-0/wal/instance:
uuid: "031d7a7279e54c6180ab5144f10a8c09"
format_stamp: "Formatted at 2026-06-20 23:37:52 on dist-test-slave-tcgx"
server_key: "425441eff8a3f4948e1540a375b69769"
server_key_iv: "ce240b27fd1380369f35bac76e766249"
server_key_version: "kuduclusterkey@0"
I20260620 23:37:52.220407 11803 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.007s sys 0.001s
I20260620 23:37:52.240622 11823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:37:52.242378 11803 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260620 23:37:52.242585 11803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/ts-0/wal
uuid: "031d7a7279e54c6180ab5144f10a8c09"
format_stamp: "Formatted at 2026-06-20 23:37:52 on dist-test-slave-tcgx"
server_key: "425441eff8a3f4948e1540a375b69769"
server_key_iv: "ce240b27fd1380369f35bac76e766249"
server_key_version: "kuduclusterkey@0"
I20260620 23:37:52.242834 11803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:37:52.281211 11803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:37:52.286770 11803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:37:52.287202 11803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:37:52.300587 11803 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:37:52.302594 11803 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:37:52.302716 11803 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:37:52.302853 11803 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:37:52.302943 11803 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:37:52.359270 11803 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44477
I20260620 23:37:52.359270 11935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44477 every 8 connection(s)
I20260620 23:37:52.360903 11803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
I20260620 23:37:52.361604 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 11803
I20260620 23:37:52.361855 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/ts-0/wal/instance
I20260620 23:37:52.362864 388 external_mini_cluster.cc:1428] Setting key 687e6bc5d289debea43f6a895f9cbd43
Jun 20 23:37:52 dist-test-slave-tcgx krb5kdc[11247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260620 23:37:52.415133 11940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:37:52.384068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55033 (local address 127.0.97.62:46603)
0620 23:37:52.389027 (+ 4959us) server_negotiation.cc:206] Beginning negotiation
0620 23:37:52.389037 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:37:52.392996 (+ 3959us) server_negotiation.cc:407] Connection header received
0620 23:37:52.393074 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:37:52.393082 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:37:52.393181 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:37:52.393288 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:37:52.395661 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:37:52.396945 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:37:52.400246 (+ 3301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:37:52.401067 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:37:52.401192 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:37:52.407941 (+ 6749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:37:52.407947 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:37:52.407950 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:37:52.407997 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:37:52.410764 (+ 2767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:37:52.411850 (+ 1086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:37:52.411859 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:37:52.411865 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:37:52.412035 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:37:52.412642 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:37:52.412651 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:37:52.412657 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:37:52.413063 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:37:52.413207 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 23:37:52.414303 (+ 1096us) server_negotiation.cc:299] Negotiation successful
0620 23:37:52.414597 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4769,"thread_start_us":4634,"threads_started":1}
I20260620 23:37:52.417971 11939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:37:52.383210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46603 (local address 127.0.97.1:55033)
0620 23:37:52.389669 (+ 6459us) negotiation.cc:107] Waiting for socket to connect
0620 23:37:52.389721 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 23:37:52.392545 (+ 2824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:37:52.393791 (+ 1246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:37:52.393807 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:37:52.394521 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:37:52.395445 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:37:52.395475 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:37:52.397178 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:37:52.397184 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:37:52.400044 (+ 2860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:37:52.400065 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:37:52.401589 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:37:52.401599 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:37:52.401983 (+ 384us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:37:52.402902 (+ 919us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:37:52.402969 (+ 67us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:37:52.407563 (+ 4594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:37:52.410967 (+ 3404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:37:52.410981 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:37:52.411005 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:37:52.411696 (+ 691us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:37:52.412193 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:37:52.412201 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:37:52.412208 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:37:52.412503 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:37:52.413301 (+ 798us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:37:52.413312 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:37:52.413832 (+ 520us) client_negotiation.cc:769] Sending connection context
0620 23:37:52.415732 (+ 1900us) client_negotiation.cc:240] Negotiation successful
0620 23:37:52.416407 (+ 675us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6134,"thread_start_us":5829,"threads_started":1}
I20260620 23:37:52.420315 11936 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46603
I20260620 23:37:52.420678 11936 heartbeater.cc:464] Registering TS with master...
I20260620 23:37:52.421571 11936 heartbeater.cc:511] Master 127.0.97.62:46603 requested a full tablet report, sending...
I20260620 23:37:52.424058 11723 ts_manager.cc:194] Registered new tserver with Master: 031d7a7279e54c6180ab5144f10a8c09 (127.0.97.1:44477)
I20260620 23:37:52.426322 11723 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:55033
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 23:37:52.446255 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:53589/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46603
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:37:52 dist-test-slave-tcgx krb5kdc[11247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998671, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 20 23:37:52 dist-test-slave-tcgx krb5kdc[11247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998671, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260620 23:37:52.913152 11942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:37:52.913669 11942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:37:52.913879 11942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:37:52.926060 11942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:37:52.926241 11942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:37:52.926513 11942 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 23:37:52.942833 11942 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:53589/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46603
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:37:52.944949 11942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:37:52.946867 11942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:37:52.971415 11949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:37:52.972477 11950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:37:52.977734 11942 server_base.cc:1034] running on GCE node
W20260620 23:37:52.977626 11953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:37:52.978996 11942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:37:52.980154 11942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:37:52.981364 11942 hybrid_clock.cc:648] HybridClock initialized: now 1781998672981298 us; error 60 us; skew 500 ppm
I20260620 23:37:52.981760 11942 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:37:52 dist-test-slave-tcgx krb5kdc[11247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:37:52.985544 11942 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 23:37:53.007972 11942 webserver.cc:460] Webserver started at http://127.0.97.2:46335/ using document root <none> and password file <none>
I20260620 23:37:53.008998 11942 fs_manager.cc:359] Metadata directory not provided
I20260620 23:37:53.009243 11942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:37:53.009769 11942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 23:37:53 dist-test-slave-tcgx krb5kdc[11247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 23:37:53.030480 11942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/ts-1/data/instance:
uuid: "80d88d00f85947c1a2d469225f9a86e5"
format_stamp: "Formatted at 2026-06-20 23:37:53 on dist-test-slave-tcgx"
server_key: "8578692eab5878c41e1ae18085fd40fc"
server_key_iv: "5bf3faa635b87f36401e6b090aa60ed8"
server_key_version: "kuduclusterkey@0"
I20260620 23:37:53.031545 11942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/ts-1/wal/instance:
uuid: "80d88d00f85947c1a2d469225f9a86e5"
format_stamp: "Formatted at 2026-06-20 23:37:53 on dist-test-slave-tcgx"
server_key: "8578692eab5878c41e1ae18085fd40fc"
server_key_iv: "5bf3faa635b87f36401e6b090aa60ed8"
server_key_version: "kuduclusterkey@0"
I20260620 23:37:53.045450 11942 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.004s sys 0.008s
I20260620 23:37:53.079921 11959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:37:53.084625 11942 fs_manager.cc:725] Time spent opening block manager: real 0.019s user 0.008s sys 0.008s
I20260620 23:37:53.084939 11942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/ts-1/wal
uuid: "80d88d00f85947c1a2d469225f9a86e5"
format_stamp: "Formatted at 2026-06-20 23:37:53 on dist-test-slave-tcgx"
server_key: "8578692eab5878c41e1ae18085fd40fc"
server_key_iv: "5bf3faa635b87f36401e6b090aa60ed8"
server_key_version: "kuduclusterkey@0"
I20260620 23:37:53.085333 11942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:37:53.134140 11942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:37:53.139940 11942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:37:53.140357 11942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:37:53.147825 11942 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:37:53.150045 11942 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:37:53.150156 11942 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:37:53.150557 11942 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:37:53.150672 11942 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260620 23:37:53.205096 11942 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43765
I20260620 23:37:53.205112 12078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43765 every 8 connection(s)
I20260620 23:37:53.206554 11942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
I20260620 23:37:53.214326 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 11942
I20260620 23:37:53.214494 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/ts-1/wal/instance
I20260620 23:37:53.215055 388 external_mini_cluster.cc:1428] Setting key af524304817252ee3430cbaaafd76ad6
Jun 20 23:37:53 dist-test-slave-tcgx krb5kdc[11247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260620 23:37:53.252792 12081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:37:53.210580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48657 (local address 127.0.97.62:46603)
0620 23:37:53.227355 (+ 16775us) server_negotiation.cc:206] Beginning negotiation
0620 23:37:53.227362 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:37:53.228505 (+ 1143us) server_negotiation.cc:407] Connection header received
0620 23:37:53.230203 (+ 1698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:37:53.230209 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:37:53.230321 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:37:53.230436 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:37:53.233719 (+ 3283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:37:53.235043 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:37:53.238270 (+ 3227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:37:53.239076 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:37:53.239202 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:37:53.244242 (+ 5040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:37:53.244253 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:37:53.244259 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:37:53.244316 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:37:53.247990 (+ 3674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:37:53.249090 (+ 1100us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:37:53.249099 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:37:53.249104 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:37:53.249250 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:37:53.249812 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:37:53.249819 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:37:53.249825 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:37:53.250223 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:37:53.250375 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 23:37:53.252188 (+ 1813us) server_negotiation.cc:299] Negotiation successful
0620 23:37:53.252353 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16629,"thread_start_us":16496,"threads_started":1}
I20260620 23:37:53.254112 12082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:37:53.210997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46603 (local address 127.0.97.2:48657)
0620 23:37:53.227898 (+ 16901us) negotiation.cc:107] Waiting for socket to connect
0620 23:37:53.227966 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0620 23:37:53.229892 (+ 1926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:37:53.231862 (+ 1970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:37:53.231877 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:37:53.232574 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:37:53.233472 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:37:53.233502 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:37:53.235260 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:37:53.235268 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:37:53.238058 (+ 2790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:37:53.238079 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:37:53.239601 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:37:53.239612 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:37:53.239963 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:37:53.240856 (+ 893us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:37:53.240899 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:37:53.243899 (+ 3000us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:37:53.248209 (+ 4310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:37:53.248220 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:37:53.248242 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:37:53.248909 (+ 667us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:37:53.249406 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:37:53.249411 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:37:53.249415 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:37:53.249666 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:37:53.250446 (+ 780us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:37:53.250452 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:37:53.250793 (+ 341us) client_negotiation.cc:769] Sending connection context
0620 23:37:53.251155 (+ 362us) client_negotiation.cc:240] Negotiation successful
0620 23:37:53.251574 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16550,"thread_start_us":16284,"threads_started":1}
I20260620 23:37:53.256264 12079 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46603
I20260620 23:37:53.256614 12079 heartbeater.cc:464] Registering TS with master...
I20260620 23:37:53.257525 12079 heartbeater.cc:511] Master 127.0.97.62:46603 requested a full tablet report, sending...
I20260620 23:37:53.259446 11723 ts_manager.cc:194] Registered new tserver with Master: 80d88d00f85947c1a2d469225f9a86e5 (127.0.97.2:43765)
I20260620 23:37:53.260614 11723 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:48657
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 23:37:53.283154 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:53589/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46603
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:37:53.430599 11936 heartbeater.cc:503] Master 127.0.97.62:46603 was elected leader, sending a full tablet report...
W20260620 23:37:53.642897 12085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:37:53.643337 12085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:37:53.643508 12085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:37:53.651268 12085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:37:53.651423 12085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:37:53.651613 12085 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 23:37:53.661974 12085 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:53589/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46603
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:37:53.663439 12085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:37:53.664773 12085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:37:53.678339 12090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:37:53.678377 12093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:37:53.678265 12091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:37:53.679540 12085 server_base.cc:1034] running on GCE node
I20260620 23:37:53.680184 12085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:37:53.681252 12085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:37:53.682514 12085 hybrid_clock.cc:648] HybridClock initialized: now 1781998673682435 us; error 44 us; skew 500 ppm
I20260620 23:37:53.682852 12085 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:37:53 dist-test-slave-tcgx krb5kdc[11247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998673, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:37:53.686092 12085 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 23:37:53.687817 12085 webserver.cc:460] Webserver started at http://127.0.97.3:45007/ using document root <none> and password file <none>
I20260620 23:37:53.688581 12085 fs_manager.cc:359] Metadata directory not provided
I20260620 23:37:53.688737 12085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:37:53.689072 12085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 23:37:53 dist-test-slave-tcgx krb5kdc[11247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998673, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 23:37:53.706275 12085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/ts-2/data/instance:
uuid: "48283a8a4c164949bdd8d8cf890aaccb"
format_stamp: "Formatted at 2026-06-20 23:37:53 on dist-test-slave-tcgx"
server_key: "99311761e93a642b4cd52dd211381d64"
server_key_iv: "f61eaf724047c85febc73bce6a3c112b"
server_key_version: "kuduclusterkey@0"
I20260620 23:37:53.707442 12085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/ts-2/wal/instance:
uuid: "48283a8a4c164949bdd8d8cf890aaccb"
format_stamp: "Formatted at 2026-06-20 23:37:53 on dist-test-slave-tcgx"
server_key: "99311761e93a642b4cd52dd211381d64"
server_key_iv: "f61eaf724047c85febc73bce6a3c112b"
server_key_version: "kuduclusterkey@0"
I20260620 23:37:53.714105 12085 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260620 23:37:53.729075 12100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:37:53.730759 12085 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:37:53.730947 12085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/ts-2/wal
uuid: "48283a8a4c164949bdd8d8cf890aaccb"
format_stamp: "Formatted at 2026-06-20 23:37:53 on dist-test-slave-tcgx"
server_key: "99311761e93a642b4cd52dd211381d64"
server_key_iv: "f61eaf724047c85febc73bce6a3c112b"
server_key_version: "kuduclusterkey@0"
I20260620 23:37:53.731196 12085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:37:53.750809 12085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:37:53.754894 12085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:37:53.755278 12085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:37:53.756639 12085 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:37:53.758018 12085 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:37:53.758116 12085 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:37:53.758258 12085 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:37:53.758327 12085 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:37:53.793902 12085 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40415
I20260620 23:37:53.793967 12212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40415 every 8 connection(s)
I20260620 23:37:53.795338 12085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
I20260620 23:37:53.797716 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 12085
I20260620 23:37:53.797905 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/ts-2/wal/instance
I20260620 23:37:53.798496 388 external_mini_cluster.cc:1428] Setting key b31b3d4bc3104e0166ff07f83b12374e
Jun 20 23:37:53 dist-test-slave-tcgx krb5kdc[11247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998673, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:37:53.816848 12216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:37:53.799445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47075 (local address 127.0.97.62:46603)
0620 23:37:53.802843 (+ 3398us) server_negotiation.cc:206] Beginning negotiation
0620 23:37:53.802847 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:37:53.802862 (+ 15us) server_negotiation.cc:407] Connection header received
0620 23:37:53.802904 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:37:53.802907 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:37:53.803044 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:37:53.803162 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:37:53.804663 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:37:53.805547 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:37:53.807200 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:37:53.807623 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:37:53.807758 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:37:53.811159 (+ 3401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:37:53.811164 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:37:53.811167 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:37:53.811199 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:37:53.813675 (+ 2476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:37:53.814433 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:37:53.814458 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:37:53.814460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:37:53.814594 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:37:53.815107 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:37:53.815112 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:37:53.815114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:37:53.815378 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:37:53.815544 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 23:37:53.816230 (+ 686us) server_negotiation.cc:299] Negotiation successful
0620 23:37:53.816463 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3290,"thread_start_us":3183,"threads_started":1}
I20260620 23:37:53.817814 12215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:37:53.798686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46603 (local address 127.0.97.3:47075)
0620 23:37:53.799822 (+ 1136us) negotiation.cc:107] Waiting for socket to connect
0620 23:37:53.799879 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 23:37:53.802259 (+ 2380us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:37:53.803430 (+ 1171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:37:53.803439 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:37:53.803871 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:37:53.804481 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:37:53.804499 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:37:53.805708 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:37:53.805713 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:37:53.807059 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:37:53.807070 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:37:53.807864 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:37:53.807870 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:37:53.808119 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:37:53.808839 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:37:53.808862 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:37:53.810877 (+ 2015us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:37:53.813822 (+ 2945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:37:53.813832 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:37:53.813850 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:37:53.814301 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:37:53.814711 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:37:53.814718 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:37:53.814721 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:37:53.814926 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:37:53.815514 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:37:53.815522 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:37:53.815870 (+ 348us) client_negotiation.cc:769] Sending connection context
0620 23:37:53.816145 (+ 275us) client_negotiation.cc:240] Negotiation successful
0620 23:37:53.816685 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":827,"thread_start_us":590,"threads_started":1}
I20260620 23:37:53.819216 12213 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46603
I20260620 23:37:53.819542 12213 heartbeater.cc:464] Registering TS with master...
I20260620 23:37:53.820281 12213 heartbeater.cc:511] Master 127.0.97.62:46603 requested a full tablet report, sending...
I20260620 23:37:53.821694 11723 ts_manager.cc:194] Registered new tserver with Master: 48283a8a4c164949bdd8d8cf890aaccb (127.0.97.3:40415)
I20260620 23:37:53.822491 11723 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:47075
I20260620 23:37:53.827527 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:37:53.844813 12216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:37:53.831985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58174 (local address 127.0.97.62:46603)
0620 23:37:53.832152 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 23:37:53.832156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:37:53.833087 (+ 931us) server_negotiation.cc:407] Connection header received
0620 23:37:53.833225 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:37:53.833228 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:37:53.833298 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:37:53.833383 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:37:53.834771 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:37:53.835625 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:37:53.836963 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:37:53.837363 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:37:53.837455 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:37:53.839892 (+ 2437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:37:53.839897 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:37:53.839901 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:37:53.839933 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:37:53.842366 (+ 2433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:37:53.843087 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:37:53.843091 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:37:53.843094 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:37:53.843213 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:37:53.843535 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:37:53.843538 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:37:53.843540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:37:53.843785 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:37:53.843963 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 23:37:53.844300 (+ 337us) server_negotiation.cc:299] Negotiation successful
0620 23:37:53.844463 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 23:37:53.849115 11723 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58174:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:37:53.856647 11723 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:37:53.889767 12228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:37:53.868933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44477 (local address 127.0.0.1:54790)
0620 23:37:53.878562 (+ 9629us) negotiation.cc:107] Waiting for socket to connect
0620 23:37:53.878605 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 23:37:53.878785 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:37:53.883044 (+ 4259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:37:53.883050 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:37:53.883074 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:37:53.883383 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:37:53.883392 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:37:53.885197 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:37:53.885202 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:37:53.887303 (+ 2101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:37:53.887313 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:37:53.888756 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:37:53.888762 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:37:53.888953 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:37:53.889035 (+ 82us) client_negotiation.cc:769] Sending connection context
0620 23:37:53.889156 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 23:37:53.889376 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9378,"thread_start_us":9211,"threads_started":1}
I20260620 23:37:53.890282 12231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:37:53.868947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40415 (local address 127.0.0.1:36392)
0620 23:37:53.879430 (+ 10483us) negotiation.cc:107] Waiting for socket to connect
0620 23:37:53.879450 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:37:53.879587 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:37:53.882552 (+ 2965us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:37:53.882556 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:37:53.882580 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:37:53.882956 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:37:53.882970 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:37:53.884777 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:37:53.884782 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:37:53.886641 (+ 1859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:37:53.886654 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:37:53.889292 (+ 2638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:37:53.889296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:37:53.889572 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:37:53.889677 (+ 105us) client_negotiation.cc:769] Sending connection context
0620 23:37:53.889761 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 23:37:53.889914 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10338,"mutex_wait_us":131,"thread_start_us":10506,"threads_started":1}
I20260620 23:37:53.890662 12227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:37:53.869230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54790 (local address 127.0.97.1:44477)
0620 23:37:53.882239 (+ 13009us) server_negotiation.cc:206] Beginning negotiation
0620 23:37:53.882249 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:37:53.882300 (+ 51us) server_negotiation.cc:407] Connection header received
0620 23:37:53.882398 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:37:53.882408 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:37:53.882675 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:37:53.882870 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:37:53.883578 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:37:53.885059 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:37:53.887454 (+ 2395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:37:53.888565 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:37:53.888728 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:37:53.888967 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0620 23:37:53.890223 (+ 1256us) server_negotiation.cc:299] Negotiation successful
0620 23:37:53.890349 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12836,"thread_start_us":12745,"threads_started":1}
I20260620 23:37:53.890749 12230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:37:53.870058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56222 (local address 127.0.97.2:43765)
0620 23:37:53.880214 (+ 10156us) server_negotiation.cc:206] Beginning negotiation
0620 23:37:53.880227 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 23:37:53.880762 (+ 535us) server_negotiation.cc:407] Connection header received
0620 23:37:53.880899 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:37:53.880910 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:37:53.881296 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:37:53.881558 (+ 262us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:37:53.882492 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:37:53.883931 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:37:53.887002 (+ 3071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:37:53.888787 (+ 1785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:37:53.888910 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:37:53.889093 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 23:37:53.890149 (+ 1056us) server_negotiation.cc:299] Negotiation successful
0620 23:37:53.890439 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9942,"thread_start_us":9778,"threads_started":1}
I20260620 23:37:53.891089 12229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:37:53.869407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36392 (local address 127.0.97.3:40415)
0620 23:37:53.881940 (+ 12533us) server_negotiation.cc:206] Beginning negotiation
0620 23:37:53.881950 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:37:53.881984 (+ 34us) server_negotiation.cc:407] Connection header received
0620 23:37:53.882078 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:37:53.882085 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:37:53.882333 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:37:53.882584 (+ 251us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:37:53.883158 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:37:53.884616 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:37:53.886783 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:37:53.888787 (+ 2004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:37:53.888910 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:37:53.889122 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0620 23:37:53.890575 (+ 1453us) server_negotiation.cc:299] Negotiation successful
0620 23:37:53.890725 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12308,"thread_start_us":12131,"threads_started":1}
I20260620 23:37:53.891407 12226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:37:53.870357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43765 (local address 127.0.0.1:56222)
0620 23:37:53.880440 (+ 10083us) negotiation.cc:107] Waiting for socket to connect
0620 23:37:53.880463 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 23:37:53.880609 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:37:53.881825 (+ 1216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:37:53.881836 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:37:53.881872 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:37:53.882300 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:37:53.882315 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:37:53.884129 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:37:53.884135 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:37:53.886813 (+ 2678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:37:53.886828 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:37:53.889278 (+ 2450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:37:53.889285 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:37:53.889570 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:37:53.889673 (+ 103us) client_negotiation.cc:769] Sending connection context
0620 23:37:53.889761 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 23:37:53.891063 (+ 1302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9924,"mutex_wait_us":1194,"thread_start_us":8792,"threads_started":1}
I20260620 23:37:53.894665 12014 tablet_service.cc:1467] Processing CreateTablet for tablet a679b7e7491844a4b8d89fec7926b78f (DEFAULT_TABLE table=test-table [id=53afcdd458a54843be38a5a0fb5badee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:37:53.894678 12148 tablet_service.cc:1467] Processing CreateTablet for tablet a679b7e7491844a4b8d89fec7926b78f (DEFAULT_TABLE table=test-table [id=53afcdd458a54843be38a5a0fb5badee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:37:53.894678 11871 tablet_service.cc:1467] Processing CreateTablet for tablet a679b7e7491844a4b8d89fec7926b78f (DEFAULT_TABLE table=test-table [id=53afcdd458a54843be38a5a0fb5badee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:37:53.896174 12014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a679b7e7491844a4b8d89fec7926b78f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:37:53.896205 12148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a679b7e7491844a4b8d89fec7926b78f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:37:53.896430 11871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a679b7e7491844a4b8d89fec7926b78f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:37:53.907850 12232 tablet_bootstrap.cc:492] T a679b7e7491844a4b8d89fec7926b78f P 80d88d00f85947c1a2d469225f9a86e5: Bootstrap starting.
I20260620 23:37:53.911157 12233 tablet_bootstrap.cc:492] T a679b7e7491844a4b8d89fec7926b78f P 031d7a7279e54c6180ab5144f10a8c09: Bootstrap starting.
I20260620 23:37:53.912848 12232 tablet_bootstrap.cc:654] T a679b7e7491844a4b8d89fec7926b78f P 80d88d00f85947c1a2d469225f9a86e5: Neither blocks nor log segments found. Creating new log.
I20260620 23:37:53.914189 12232 log.cc:824] T a679b7e7491844a4b8d89fec7926b78f P 80d88d00f85947c1a2d469225f9a86e5: Log is configured to *not* fsync() on all Append() calls
I20260620 23:37:53.914808 12233 tablet_bootstrap.cc:654] T a679b7e7491844a4b8d89fec7926b78f P 031d7a7279e54c6180ab5144f10a8c09: Neither blocks nor log segments found. Creating new log.
I20260620 23:37:53.916345 12233 log.cc:824] T a679b7e7491844a4b8d89fec7926b78f P 031d7a7279e54c6180ab5144f10a8c09: Log is configured to *not* fsync() on all Append() calls
I20260620 23:37:53.917809 12234 tablet_bootstrap.cc:492] T a679b7e7491844a4b8d89fec7926b78f P 48283a8a4c164949bdd8d8cf890aaccb: Bootstrap starting.
I20260620 23:37:53.921169 12234 tablet_bootstrap.cc:654] T a679b7e7491844a4b8d89fec7926b78f P 48283a8a4c164949bdd8d8cf890aaccb: Neither blocks nor log segments found. Creating new log.
I20260620 23:37:53.921204 12233 tablet_bootstrap.cc:492] T a679b7e7491844a4b8d89fec7926b78f P 031d7a7279e54c6180ab5144f10a8c09: No bootstrap required, opened a new log
I20260620 23:37:53.921204 12232 tablet_bootstrap.cc:492] T a679b7e7491844a4b8d89fec7926b78f P 80d88d00f85947c1a2d469225f9a86e5: No bootstrap required, opened a new log
I20260620 23:37:53.921489 12233 ts_tablet_manager.cc:1397] T a679b7e7491844a4b8d89fec7926b78f P 031d7a7279e54c6180ab5144f10a8c09: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20260620 23:37:53.921489 12232 ts_tablet_manager.cc:1397] T a679b7e7491844a4b8d89fec7926b78f P 80d88d00f85947c1a2d469225f9a86e5: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.005s
I20260620 23:37:53.922320 12234 log.cc:824] T a679b7e7491844a4b8d89fec7926b78f P 48283a8a4c164949bdd8d8cf890aaccb: Log is configured to *not* fsync() on all Append() calls
I20260620 23:37:53.925729 12234 tablet_bootstrap.cc:492] T a679b7e7491844a4b8d89fec7926b78f P 48283a8a4c164949bdd8d8cf890aaccb: No bootstrap required, opened a new log
I20260620 23:37:53.926009 12234 ts_tablet_manager.cc:1397] T a679b7e7491844a4b8d89fec7926b78f P 48283a8a4c164949bdd8d8cf890aaccb: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 23:37:53.933336 12234 raft_consensus.cc:348] T a679b7e7491844a4b8d89fec7926b78f P 48283a8a4c164949bdd8d8cf890aaccb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031d7a7279e54c6180ab5144f10a8c09" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44477 } } peers { permanent_uuid: "48283a8a4c164949bdd8d8cf890aaccb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40415 } } peers { permanent_uuid: "80d88d00f85947c1a2d469225f9a86e5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43765 } }
I20260620 23:37:53.933213 12232 raft_consensus.cc:348] T a679b7e7491844a4b8d89fec7926b78f P 80d88d00f85947c1a2d469225f9a86e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031d7a7279e54c6180ab5144f10a8c09" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44477 } } peers { permanent_uuid: "48283a8a4c164949bdd8d8cf890aaccb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40415 } } peers { permanent_uuid: "80d88d00f85947c1a2d469225f9a86e5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43765 } }
I20260620 23:37:53.933207 12233 raft_consensus.cc:348] T a679b7e7491844a4b8d89fec7926b78f P 031d7a7279e54c6180ab5144f10a8c09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031d7a7279e54c6180ab5144f10a8c09" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44477 } } peers { permanent_uuid: "48283a8a4c164949bdd8d8cf890aaccb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40415 } } peers { permanent_uuid: "80d88d00f85947c1a2d469225f9a86e5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43765 } }
I20260620 23:37:53.933681 12233 raft_consensus.cc:374] T a679b7e7491844a4b8d89fec7926b78f P 031d7a7279e54c6180ab5144f10a8c09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:37:53.933681 12232 raft_consensus.cc:374] T a679b7e7491844a4b8d89fec7926b78f P 80d88d00f85947c1a2d469225f9a86e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:37:53.933693 12234 raft_consensus.cc:374] T a679b7e7491844a4b8d89fec7926b78f P 48283a8a4c164949bdd8d8cf890aaccb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:37:53.933780 12234 raft_consensus.cc:729] T a679b7e7491844a4b8d89fec7926b78f P 48283a8a4c164949bdd8d8cf890aaccb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48283a8a4c164949bdd8d8cf890aaccb, State: Initialized, Role: FOLLOWER
I20260620 23:37:53.933784 12232 raft_consensus.cc:729] T a679b7e7491844a4b8d89fec7926b78f P 80d88d00f85947c1a2d469225f9a86e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80d88d00f85947c1a2d469225f9a86e5, State: Initialized, Role: FOLLOWER
I20260620 23:37:53.933784 12233 raft_consensus.cc:729] T a679b7e7491844a4b8d89fec7926b78f P 031d7a7279e54c6180ab5144f10a8c09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031d7a7279e54c6180ab5144f10a8c09, State: Initialized, Role: FOLLOWER
I20260620 23:37:53.934427 12234 consensus_queue.cc:260] T a679b7e7491844a4b8d89fec7926b78f P 48283a8a4c164949bdd8d8cf890aaccb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031d7a7279e54c6180ab5144f10a8c09" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44477 } } peers { permanent_uuid: "48283a8a4c164949bdd8d8cf890aaccb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40415 } } peers { permanent_uuid: "80d88d00f85947c1a2d469225f9a86e5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43765 } }
I20260620 23:37:53.934476 12232 consensus_queue.cc:260] T a679b7e7491844a4b8d89fec7926b78f P 80d88d00f85947c1a2d469225f9a86e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031d7a7279e54c6180ab5144f10a8c09" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44477 } } peers { permanent_uuid: "48283a8a4c164949bdd8d8cf890aaccb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40415 } } peers { permanent_uuid: "80d88d00f85947c1a2d469225f9a86e5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43765 } }
I20260620 23:37:53.934476 12233 consensus_queue.cc:260] T a679b7e7491844a4b8d89fec7926b78f P 031d7a7279e54c6180ab5144f10a8c09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031d7a7279e54c6180ab5144f10a8c09" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44477 } } peers { permanent_uuid: "48283a8a4c164949bdd8d8cf890aaccb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40415 } } peers { permanent_uuid: "80d88d00f85947c1a2d469225f9a86e5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43765 } }
I20260620 23:37:53.936308 12234 ts_tablet_manager.cc:1428] T a679b7e7491844a4b8d89fec7926b78f P 48283a8a4c164949bdd8d8cf890aaccb: Time spent starting tablet: real 0.010s user 0.004s sys 0.007s
I20260620 23:37:53.937207 12213 heartbeater.cc:503] Master 127.0.97.62:46603 was elected leader, sending a full tablet report...
I20260620 23:37:53.938354 12232 ts_tablet_manager.cc:1428] T a679b7e7491844a4b8d89fec7926b78f P 80d88d00f85947c1a2d469225f9a86e5: Time spent starting tablet: real 0.017s user 0.013s sys 0.005s
I20260620 23:37:53.938634 12233 ts_tablet_manager.cc:1428] T a679b7e7491844a4b8d89fec7926b78f P 031d7a7279e54c6180ab5144f10a8c09: Time spent starting tablet: real 0.017s user 0.010s sys 0.008s
I20260620 23:37:53.938637 12079 heartbeater.cc:503] Master 127.0.97.62:46603 was elected leader, sending a full tablet report...
W20260620 23:37:53.960805 12080 tablet.cc:2369] T a679b7e7491844a4b8d89fec7926b78f P 80d88d00f85947c1a2d469225f9a86e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:37:54.047201 12214 tablet.cc:2369] T a679b7e7491844a4b8d89fec7926b78f P 48283a8a4c164949bdd8d8cf890aaccb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:37:54.062160 12238 raft_consensus.cc:482] T a679b7e7491844a4b8d89fec7926b78f P 48283a8a4c164949bdd8d8cf890aaccb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:37:54.062485 12238 raft_consensus.cc:504] T a679b7e7491844a4b8d89fec7926b78f P 48283a8a4c164949bdd8d8cf890aaccb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031d7a7279e54c6180ab5144f10a8c09" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44477 } } peers { permanent_uuid: "48283a8a4c164949bdd8d8cf890aaccb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40415 } } peers { permanent_uuid: "80d88d00f85947c1a2d469225f9a86e5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43765 } }
I20260620 23:37:54.064416 12238 leader_election.cc:288] T a679b7e7491844a4b8d89fec7926b78f P 48283a8a4c164949bdd8d8cf890aaccb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 031d7a7279e54c6180ab5144f10a8c09 (127.0.97.1:44477), 80d88d00f85947c1a2d469225f9a86e5 (127.0.97.2:43765)
I20260620 23:37:54.070593 12215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:37:54.064656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44477 (local address 127.0.97.3:37961)
0620 23:37:54.064919 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 23:37:54.064940 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:37:54.065076 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:37:54.065641 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:37:54.065646 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:37:54.065671 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:37:54.065900 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:37:54.065908 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:37:54.067411 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:37:54.067416 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:37:54.068731 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:37:54.068741 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:37:54.069832 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:37:54.069837 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:37:54.070031 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:37:54.070055 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 23:37:54.070162 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 23:37:54.070303 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"spinlock_wait_cycles":9984}
I20260620 23:37:54.070587 12227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:37:54.065058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37961 (local address 127.0.97.1:44477)
0620 23:37:54.065308 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 23:37:54.065317 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:37:54.065339 (+ 22us) server_negotiation.cc:407] Connection header received
0620 23:37:54.065413 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:37:54.065418 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:37:54.065491 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:37:54.065613 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:37:54.066044 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:37:54.067270 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:37:54.068898 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:37:54.069677 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:37:54.069824 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:37:54.069928 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 23:37:54.070188 (+ 260us) server_negotiation.cc:299] Negotiation successful
0620 23:37:54.070314 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 23:37:54.070978 12230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:37:54.064950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58623 (local address 127.0.97.2:43765)
0620 23:37:54.065179 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 23:37:54.065187 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:37:54.065655 (+ 468us) server_negotiation.cc:407] Connection header received
0620 23:37:54.065876 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:37:54.065890 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:37:54.065947 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:37:54.066048 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:37:54.066878 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:37:54.067648 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:37:54.069266 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:37:54.070044 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:37:54.070201 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:37:54.070295 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 23:37:54.070578 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 23:37:54.070700 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260620 23:37:54.071096 12241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:37:54.064752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43765 (local address 127.0.97.3:58623)
0620 23:37:54.065549 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0620 23:37:54.065562 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 23:37:54.065732 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:37:54.066073 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:37:54.066078 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:37:54.066100 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:37:54.066728 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:37:54.066736 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:37:54.067794 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:37:54.067801 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:37:54.069119 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:37:54.069129 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:37:54.070248 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:37:54.070254 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:37:54.070409 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:37:54.070432 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 23:37:54.070550 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 23:37:54.070677 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":705,"spinlock_wait_cycles":7168,"thread_start_us":537,"threads_started":1}
I20260620 23:37:54.071810 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a679b7e7491844a4b8d89fec7926b78f" candidate_uuid: "48283a8a4c164949bdd8d8cf890aaccb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031d7a7279e54c6180ab5144f10a8c09" is_pre_election: true
I20260620 23:37:54.072070 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a679b7e7491844a4b8d89fec7926b78f" candidate_uuid: "48283a8a4c164949bdd8d8cf890aaccb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80d88d00f85947c1a2d469225f9a86e5" is_pre_election: true
I20260620 23:37:54.072376 11891 raft_consensus.cc:2454] T a679b7e7491844a4b8d89fec7926b78f P 031d7a7279e54c6180ab5144f10a8c09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48283a8a4c164949bdd8d8cf890aaccb in term 0.
I20260620 23:37:54.072592 12034 raft_consensus.cc:2454] T a679b7e7491844a4b8d89fec7926b78f P 80d88d00f85947c1a2d469225f9a86e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48283a8a4c164949bdd8d8cf890aaccb in term 0.
I20260620 23:37:54.073145 12102 leader_election.cc:302] T a679b7e7491844a4b8d89fec7926b78f P 48283a8a4c164949bdd8d8cf890aaccb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031d7a7279e54c6180ab5144f10a8c09, 48283a8a4c164949bdd8d8cf890aaccb; no voters:
I20260620 23:37:54.073559 12238 raft_consensus.cc:2781] T a679b7e7491844a4b8d89fec7926b78f P 48283a8a4c164949bdd8d8cf890aaccb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:37:54.073704 12238 raft_consensus.cc:482] T a679b7e7491844a4b8d89fec7926b78f P 48283a8a4c164949bdd8d8cf890aaccb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:37:54.073825 12238 raft_consensus.cc:3037] T a679b7e7491844a4b8d89fec7926b78f P 48283a8a4c164949bdd8d8cf890aaccb [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:37:54.076094 12238 raft_consensus.cc:504] T a679b7e7491844a4b8d89fec7926b78f P 48283a8a4c164949bdd8d8cf890aaccb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031d7a7279e54c6180ab5144f10a8c09" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44477 } } peers { permanent_uuid: "48283a8a4c164949bdd8d8cf890aaccb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40415 } } peers { permanent_uuid: "80d88d00f85947c1a2d469225f9a86e5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43765 } }
I20260620 23:37:54.076678 12238 leader_election.cc:288] T a679b7e7491844a4b8d89fec7926b78f P 48283a8a4c164949bdd8d8cf890aaccb [CANDIDATE]: Term 1 election: Requested vote from peers 031d7a7279e54c6180ab5144f10a8c09 (127.0.97.1:44477), 80d88d00f85947c1a2d469225f9a86e5 (127.0.97.2:43765)
I20260620 23:37:54.077244 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a679b7e7491844a4b8d89fec7926b78f" candidate_uuid: "48283a8a4c164949bdd8d8cf890aaccb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80d88d00f85947c1a2d469225f9a86e5"
I20260620 23:37:54.077670 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a679b7e7491844a4b8d89fec7926b78f" candidate_uuid: "48283a8a4c164949bdd8d8cf890aaccb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031d7a7279e54c6180ab5144f10a8c09"
I20260620 23:37:54.077864 11891 raft_consensus.cc:3037] T a679b7e7491844a4b8d89fec7926b78f P 031d7a7279e54c6180ab5144f10a8c09 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:37:54.077862 12034 raft_consensus.cc:3037] T a679b7e7491844a4b8d89fec7926b78f P 80d88d00f85947c1a2d469225f9a86e5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:37:54.081086 11891 raft_consensus.cc:2454] T a679b7e7491844a4b8d89fec7926b78f P 031d7a7279e54c6180ab5144f10a8c09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48283a8a4c164949bdd8d8cf890aaccb in term 1.
I20260620 23:37:54.081086 12034 raft_consensus.cc:2454] T a679b7e7491844a4b8d89fec7926b78f P 80d88d00f85947c1a2d469225f9a86e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48283a8a4c164949bdd8d8cf890aaccb in term 1.
I20260620 23:37:54.081614 12102 leader_election.cc:302] T a679b7e7491844a4b8d89fec7926b78f P 48283a8a4c164949bdd8d8cf890aaccb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031d7a7279e54c6180ab5144f10a8c09, 48283a8a4c164949bdd8d8cf890aaccb; no voters:
I20260620 23:37:54.081951 12238 raft_consensus.cc:2781] T a679b7e7491844a4b8d89fec7926b78f P 48283a8a4c164949bdd8d8cf890aaccb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:37:54.082408 12238 raft_consensus.cc:686] T a679b7e7491844a4b8d89fec7926b78f P 48283a8a4c164949bdd8d8cf890aaccb [term 1 LEADER]: Becoming Leader. State: Replica: 48283a8a4c164949bdd8d8cf890aaccb, State: Running, Role: LEADER
I20260620 23:37:54.082891 12238 consensus_queue.cc:237] T a679b7e7491844a4b8d89fec7926b78f P 48283a8a4c164949bdd8d8cf890aaccb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031d7a7279e54c6180ab5144f10a8c09" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44477 } } peers { permanent_uuid: "48283a8a4c164949bdd8d8cf890aaccb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40415 } } peers { permanent_uuid: "80d88d00f85947c1a2d469225f9a86e5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43765 } }
I20260620 23:37:54.087636 11721 catalog_manager.cc:5495] T a679b7e7491844a4b8d89fec7926b78f P 48283a8a4c164949bdd8d8cf890aaccb reported cstate change: term changed from 0 to 1, leader changed from <none> to 48283a8a4c164949bdd8d8cf890aaccb (127.0.97.3). New cstate: current_term: 1 leader_uuid: "48283a8a4c164949bdd8d8cf890aaccb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031d7a7279e54c6180ab5144f10a8c09" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48283a8a4c164949bdd8d8cf890aaccb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80d88d00f85947c1a2d469225f9a86e5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43765 } health_report { overall_health: UNKNOWN } } }
I20260620 23:37:54.123487 12229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:37:54.117606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36406 (local address 127.0.97.3:40415)
0620 23:37:54.117810 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 23:37:54.117817 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:37:54.117836 (+ 19us) server_negotiation.cc:407] Connection header received
0620 23:37:54.117992 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:37:54.117996 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:37:54.118073 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:37:54.118231 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:37:54.118609 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:37:54.119801 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:37:54.121295 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:37:54.121693 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:37:54.121829 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:37:54.122369 (+ 540us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:37:54.122884 (+ 515us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:37:54.123089 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 23:37:54.123144 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 23:37:54.123223 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 23:37:54.131946 12034 raft_consensus.cc:1261] T a679b7e7491844a4b8d89fec7926b78f P 80d88d00f85947c1a2d469225f9a86e5 [term 1 FOLLOWER]: Refusing update from remote peer 48283a8a4c164949bdd8d8cf890aaccb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:37:54.132019 11891 raft_consensus.cc:1261] T a679b7e7491844a4b8d89fec7926b78f P 031d7a7279e54c6180ab5144f10a8c09 [term 1 FOLLOWER]: Refusing update from remote peer 48283a8a4c164949bdd8d8cf890aaccb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:37:54.133026 12242 consensus_queue.cc:1035] T a679b7e7491844a4b8d89fec7926b78f P 48283a8a4c164949bdd8d8cf890aaccb [LEADER]: Connected to new peer: Peer: permanent_uuid: "80d88d00f85947c1a2d469225f9a86e5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:37:54.133399 12238 consensus_queue.cc:1035] T a679b7e7491844a4b8d89fec7926b78f P 48283a8a4c164949bdd8d8cf890aaccb [LEADER]: Connected to new peer: Peer: permanent_uuid: "031d7a7279e54c6180ab5144f10a8c09" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 23:37:54.138080 11938 tablet.cc:2369] T a679b7e7491844a4b8d89fec7926b78f P 031d7a7279e54c6180ab5144f10a8c09: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:37:54.149035 12247 mvcc.cc:205] Tried to move back new op lower bound from 7299066569234432000 to 7299066569047334912. Current Snapshot: MvccSnapshot[applied={T|T < 7299066569234432000}]
I20260620 23:37:54.165552 11722 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:37:54.165899 11722 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:37:54.169919 11722 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 88e3f05172b14287ae49b3a58f67f829: Sending DeleteTablet for 3 replicas of tablet a679b7e7491844a4b8d89fec7926b78f
I20260620 23:37:54.171126 12148 tablet_service.cc:1514] Processing DeleteTablet for tablet a679b7e7491844a4b8d89fec7926b78f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:37:54 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36392
I20260620 23:37:54.171214 11871 tablet_service.cc:1514] Processing DeleteTablet for tablet a679b7e7491844a4b8d89fec7926b78f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:37:54 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54790
I20260620 23:37:54.171859 12014 tablet_service.cc:1514] Processing DeleteTablet for tablet a679b7e7491844a4b8d89fec7926b78f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:37:54 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56222
I20260620 23:37:54.172152 12255 tablet_replica.cc:330] T a679b7e7491844a4b8d89fec7926b78f P 48283a8a4c164949bdd8d8cf890aaccb: stopping tablet replica
I20260620 23:37:54.173682 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 11803
I20260620 23:37:54.173969 12255 raft_consensus.cc:2229] T a679b7e7491844a4b8d89fec7926b78f P 48283a8a4c164949bdd8d8cf890aaccb [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:37:54.174778 12255 raft_consensus.cc:2258] T a679b7e7491844a4b8d89fec7926b78f P 48283a8a4c164949bdd8d8cf890aaccb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:37:54.176829 12255 ts_tablet_manager.cc:1905] T a679b7e7491844a4b8d89fec7926b78f P 48283a8a4c164949bdd8d8cf890aaccb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:37:54.186589 12255 ts_tablet_manager.cc:1918] T a679b7e7491844a4b8d89fec7926b78f P 48283a8a4c164949bdd8d8cf890aaccb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:37:54.186921 12255 log.cc:1196] T a679b7e7491844a4b8d89fec7926b78f P 48283a8a4c164949bdd8d8cf890aaccb: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/ts-2/wal/wals/a679b7e7491844a4b8d89fec7926b78f
I20260620 23:37:54.187661 12255 ts_tablet_manager.cc:1939] T a679b7e7491844a4b8d89fec7926b78f P 48283a8a4c164949bdd8d8cf890aaccb: Deleting consensus metadata
I20260620 23:37:54.189657 11707 catalog_manager.cc:4841] TS 48283a8a4c164949bdd8d8cf890aaccb (127.0.97.3:40415): tablet a679b7e7491844a4b8d89fec7926b78f (table test-table [id=53afcdd458a54843be38a5a0fb5badee]) successfully deleted
I20260620 23:37:54.191308 12256 tablet_replica.cc:330] T a679b7e7491844a4b8d89fec7926b78f P 031d7a7279e54c6180ab5144f10a8c09: stopping tablet replica
I20260620 23:37:54.191807 12256 raft_consensus.cc:2229] T a679b7e7491844a4b8d89fec7926b78f P 031d7a7279e54c6180ab5144f10a8c09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:37:54.192268 12256 raft_consensus.cc:2258] T a679b7e7491844a4b8d89fec7926b78f P 031d7a7279e54c6180ab5144f10a8c09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:37:54.193089 12257 tablet_replica.cc:330] T a679b7e7491844a4b8d89fec7926b78f P 80d88d00f85947c1a2d469225f9a86e5: stopping tablet replica
I20260620 23:37:54.193560 12257 raft_consensus.cc:2229] T a679b7e7491844a4b8d89fec7926b78f P 80d88d00f85947c1a2d469225f9a86e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:37:54.194085 12257 raft_consensus.cc:2258] T a679b7e7491844a4b8d89fec7926b78f P 80d88d00f85947c1a2d469225f9a86e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:37:54.194681 12256 ts_tablet_manager.cc:1905] T a679b7e7491844a4b8d89fec7926b78f P 031d7a7279e54c6180ab5144f10a8c09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:37:54.196341 12257 ts_tablet_manager.cc:1905] T a679b7e7491844a4b8d89fec7926b78f P 80d88d00f85947c1a2d469225f9a86e5: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 23:37:54 dist-test-slave-tcgx krb5kdc[11247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 23:37:54.202455 12256 ts_tablet_manager.cc:1918] T a679b7e7491844a4b8d89fec7926b78f P 031d7a7279e54c6180ab5144f10a8c09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:37:54.202638 12256 log.cc:1196] T a679b7e7491844a4b8d89fec7926b78f P 031d7a7279e54c6180ab5144f10a8c09: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/ts-0/wal/wals/a679b7e7491844a4b8d89fec7926b78f
I20260620 23:37:54.203191 12256 ts_tablet_manager.cc:1939] T a679b7e7491844a4b8d89fec7926b78f P 031d7a7279e54c6180ab5144f10a8c09: Deleting consensus metadata
I20260620 23:37:54.203590 12257 ts_tablet_manager.cc:1918] T a679b7e7491844a4b8d89fec7926b78f P 80d88d00f85947c1a2d469225f9a86e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:37:54.203738 12257 log.cc:1196] T a679b7e7491844a4b8d89fec7926b78f P 80d88d00f85947c1a2d469225f9a86e5: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998525501163-388-0/minicluster-data/ts-1/wal/wals/a679b7e7491844a4b8d89fec7926b78f
I20260620 23:37:54.205917 12257 ts_tablet_manager.cc:1939] T a679b7e7491844a4b8d89fec7926b78f P 80d88d00f85947c1a2d469225f9a86e5: Deleting consensus metadata
I20260620 23:37:54.207911 12227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:37:54.175083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54804 (local address 127.0.97.1:44477)
0620 23:37:54.175258 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 23:37:54.175264 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:37:54.185088 (+ 9824us) server_negotiation.cc:407] Connection header received
0620 23:37:54.185606 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:37:54.185611 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:37:54.185721 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:37:54.185794 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:37:54.187808 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:37:54.188955 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:37:54.190752 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:37:54.191564 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:37:54.191692 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:37:54.200044 (+ 8352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:37:54.200063 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:37:54.200081 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:37:54.200124 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:37:54.204173 (+ 4049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:37:54.205218 (+ 1045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:37:54.205228 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:37:54.205231 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:37:54.205406 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:37:54.205865 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:37:54.205871 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:37:54.205876 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:37:54.206357 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:37:54.206623 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0620 23:37:54.207322 (+ 699us) server_negotiation.cc:299] Negotiation successful
0620 23:37:54.207541 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 23:37:54.207899 11710 catalog_manager.cc:4841] TS 80d88d00f85947c1a2d469225f9a86e5 (127.0.97.2:43765): tablet a679b7e7491844a4b8d89fec7926b78f (table test-table [id=53afcdd458a54843be38a5a0fb5badee]) successfully deleted
I20260620 23:37:54.208709 11708 catalog_manager.cc:4841] TS 031d7a7279e54c6180ab5144f10a8c09 (127.0.97.1:44477): tablet a679b7e7491844a4b8d89fec7926b78f (table test-table [id=53afcdd458a54843be38a5a0fb5badee]) successfully deleted
I20260620 23:37:54.209483 11931 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:37:54.302142 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 11803
I20260620 23:37:54.326220 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 11942
Jun 20 23:37:54 dist-test-slave-tcgx krb5kdc[11247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 23:37:54.342478 12230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:37:54.327127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56234 (local address 127.0.97.2:43765)
0620 23:37:54.327297 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 23:37:54.327302 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:37:54.327463 (+ 161us) server_negotiation.cc:407] Connection header received
0620 23:37:54.327695 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:37:54.327701 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:37:54.327787 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:37:54.327896 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:37:54.329140 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:37:54.329912 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:37:54.331318 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:37:54.331730 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:37:54.331869 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:37:54.336192 (+ 4323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:37:54.336214 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:37:54.336230 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:37:54.336264 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:37:54.339394 (+ 3130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:37:54.340284 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:37:54.340292 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:37:54.340294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:37:54.340436 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:37:54.340875 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:37:54.340879 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:37:54.340881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:37:54.341255 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:37:54.341550 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0620 23:37:54.341889 (+ 339us) server_negotiation.cc:299] Negotiation successful
0620 23:37:54.342133 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 23:37:54.343255 12074 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:37:54.433070 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 11942
I20260620 23:37:54.450033 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 12085
Jun 20 23:37:54 dist-test-slave-tcgx krb5kdc[11247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 23:37:54.471084 12229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:37:54.450898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36418 (local address 127.0.97.3:40415)
0620 23:37:54.451099 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 23:37:54.451104 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:37:54.451278 (+ 174us) server_negotiation.cc:407] Connection header received
0620 23:37:54.451420 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:37:54.451428 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:37:54.451520 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:37:54.451653 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:37:54.452942 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:37:54.454198 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:37:54.455675 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:37:54.456351 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:37:54.456493 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:37:54.463888 (+ 7395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:37:54.463907 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:37:54.463928 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:37:54.463984 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:37:54.468042 (+ 4058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:37:54.468773 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:37:54.468784 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:37:54.468789 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:37:54.468977 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:37:54.469343 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:37:54.469348 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:37:54.469351 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:37:54.469861 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:37:54.470215 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0620 23:37:54.470389 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 23:37:54.470636 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 23:37:54.471959 12208 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:37:54.612722 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 12085
I20260620 23:37:54.632062 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 11691
I20260620 23:37:54.633273 11753 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:37:54.745100 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 11691
I20260620 23:37:54.769912 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 23:37:55.105788 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 23:37:55.106076 388 mini_ranger.cc:66] Stopping Ranger...
I20260620 23:37:55.452066 388 mini_ranger.cc:68] Stopped Ranger
2026-06-20 23:37:55.452 UTC [11277] LOG: received smart shutdown request
2026-06-20 23:37:55.455 UTC [11277] LOG: background worker "logical replication launcher" (PID 11286) exited with exit code 1
2026-06-20 23:37:55.455 UTC [11281] LOG: shutting down
2026-06-20 23:37:55.625 UTC [11277] LOG: database system is shut down
2026-06-20T23:37:55Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49582 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:37:55 dist-test-slave-tcgx krb5kdc[12271](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:37:55 dist-test-slave-tcgx krb5kdc[12271](info): set up 2 sockets
Jun 20 23:37:55 dist-test-slave-tcgx krb5kdc[12271](info): commencing operation
krb5kdc: starting...
W20260620 23:37:56.875453 388 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/krb5kdc/test-user.keytab.
Jun 20 23:37:57 dist-test-slave-tcgx krb5kdc[12271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:37:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:37:57Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260620 23:37:57.210800 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskN6eSV7/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/postgres -l logfile start
2026-06-20 23:37:59.836 UTC [12301] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 23:37:59.836 UTC [12301] LOG: listening on IPv4 address "127.0.97.20", port 33393
2026-06-20 23:37:59.840 UTC [12301] LOG: listening on Unix socket "/tmp/.s.PGSQL.33393"
2026-06-20 23:37:59.862 UTC [12304] LOG: database system was shut down at 2026-06-20 23:37:58 UTC
2026-06-20 23:37:59.866 UTC [12301] LOG: database system is ready to accept connections
I20260620 23:38:00.843068 388 mini_postgres.cc:96] Postgres bound to 33393
2026-06-20 23:38:00.860 UTC [12312] FATAL: database "slave" does not exist
127.0.97.20:33393 - accepting connections
I20260620 23:38:00.861286 388 mini_ranger.cc:161] Starting Ranger...
I20260620 23:38:00.892228 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 23:38:01.404744 388 mini_ranger.cc:87] Created ranger Postgres database
I20260620 23:38:01.404877 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ranger-admin
2026-06-20 23:38:01,720 [I] DB FLAVOR :POSTGRES
2026-06-20 23:38:01,721 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 23:38:01,721 [I] Checking connection..
2026-06-20 23:38:01,721 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN6eSV7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN6eSV7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33393/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 23:38:01,956 [I] Checking connection passed.
2026-06-20 23:38:01,956 [I] --------- Verifying version history table ---------
2026-06-20 23:38:01,956 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN6eSV7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN6eSV7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33393/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 23:38:02,198 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 23:38:02,199 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN6eSV7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN6eSV7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33393/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 23:38:02,453 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 23:38:02,453 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 23:38:02,454 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN6eSV7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN6eSV7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33393/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskN6eSV7/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 23:38:02.722 UTC [12376] WARNING: there is no transaction in progress
2026-06-20 23:38:02,729 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 23:38:02,729 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN6eSV7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN6eSV7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33393/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 23:38:02,965 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 23:38:02,965 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 23:38:02,965 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN6eSV7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN6eSV7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33393/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 23:38:03,207 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN6eSV7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN6eSV7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33393/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 23:38:03,473 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN6eSV7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN6eSV7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33393/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-tcgx.c.gcp-upstream.internal','N') ;"
2026-06-20 23:38:03,711 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN6eSV7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN6eSV7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33393/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 23:38:03,977 [I] Table x_portal_user does not exist in database ranger
2026-06-20 23:38:03,978 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN6eSV7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN6eSV7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33393/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 23:38:04,236 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 23:38:04,237 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN6eSV7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN6eSV7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33393/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 23:38:04,472 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN6eSV7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN6eSV7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33393/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 23:38:04,732 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 23:38:04,732 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN6eSV7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN6eSV7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33393/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskN6eSV7/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 23:38:05.748 UTC [12484] WARNING: there is no transaction in progress
2026-06-20 23:38:05.765 UTC [12484] WARNING: there is no transaction in progress
2026-06-20 23:38:05.782 UTC [12484] WARNING: there is no transaction in progress
2026-06-20 23:38:05.798 UTC [12484] WARNING: there is no transaction in progress
2026-06-20 23:38:05.815 UTC [12484] WARNING: there is no transaction in progress
2026-06-20 23:38:05.834 UTC [12484] WARNING: there is no transaction in progress
2026-06-20 23:38:05.893 UTC [12484] WARNING: there is no transaction in progress
2026-06-20 23:38:05.900 UTC [12484] WARNING: there is no transaction in progress
2026-06-20 23:38:05.915 UTC [12484] WARNING: there is no transaction in progress
2026-06-20 23:38:05.926 UTC [12484] WARNING: there is no transaction in progress
2026-06-20 23:38:05.937 UTC [12484] WARNING: there is no transaction in progress
2026-06-20 23:38:05.952 UTC [12484] WARNING: there is no transaction in progress
2026-06-20 23:38:05.962 UTC [12484] WARNING: there is no transaction in progress
2026-06-20 23:38:05.969 UTC [12484] WARNING: there is no transaction in progress
2026-06-20 23:38:05.977 UTC [12484] WARNING: there is no transaction in progress
2026-06-20 23:38:05.985 UTC [12484] WARNING: there is no transaction in progress
2026-06-20 23:38:06,669 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 23:38:06,670 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN6eSV7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN6eSV7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33393/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 23:38:06,905 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 23:38:06,906 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN6eSV7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN6eSV7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33393/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 23:38:07,142 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 23:38:07,142 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN6eSV7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN6eSV7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33393/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 23:38:07,408 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 23:38:07,409 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN6eSV7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN6eSV7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33393/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 23:38:07,666 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN6eSV7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN6eSV7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33393/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 23:38:07,932 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN6eSV7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN6eSV7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33393/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-tcgx.c.gcp-upstream.internal';"
2026-06-20 23:38:08,159 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 23:38:08,159 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN6eSV7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN6eSV7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33393/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 23:38:08,412 [I] DB_PATCHES have already been applied
I20260620 23:38:08.419078 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskN6eSV7/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskN6eSV7/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskN6eSV7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN6eSV7/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 23:38:08.419175 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260620 23:38:08.434355 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:37693
Jun 20, 2026 11:38:08 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 11:38:08 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 11:38:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 11:38:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 11:38:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 11:38:09 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 11:38:09 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 23:38:09.596834 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.736s user 0.003s sys 0.042s
Jun 20, 2026 11:38:29 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 11:38:29 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 11:38:29 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 11:38:29 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 11:38:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 11:38:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 11:38:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 11:38:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 11:38:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 11:38:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 11:38:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 11:38:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 11:38:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 11:38:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 11:38:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 11:38:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 11:38:29 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 11:38:30 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 23:38:30.947232 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 23:38:31 dist-test-slave-tcgx krb5kdc[12271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998711, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 23:38:31.147799 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 23:38:31.171602 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 23:38:32.144971 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 23:38:32.145087 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ranger-kms
I20260620 23:38:32.145138 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:33393
2026-06-20 23:38:32,492 [I] DB FLAVOR :POSTGRES
2026-06-20 23:38:32,492 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 23:38:32,492 [I] Checking connection
2026-06-20 23:38:32,492 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN6eSV7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33393/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 23:38:32,739 [I] connection success
2026-06-20 23:38:32,739 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 23:38:32,740 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 23:38:32,740 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN6eSV7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33393/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 23:38:32,970 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 23:38:32,971 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 23:38:32,971 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 23:38:32,971 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskN6eSV7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33393/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 23:38:33,228 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 23:38:33.752040 388 mini_ranger_kms.cc:310] Created kms service
I20260620 23:38:33.973255 388 mini_ranger_kms.cc:326] Created kudu user
I20260620 23:38:34.051457 388 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 23:38:34 dist-test-slave-tcgx krb5kdc[12271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998711, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 23:38:34.511152 388 mini_ranger_kms.cc:384] Added ranger policy
I20260620 23:38:34.511466 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskN6eSV7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskN6eSV7/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskN6eSV7/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 23:38:34.511509 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260620 23:38:34.532292 388 mini_ranger_kms.cc:276] Ranger KMS PID: 12664
I20260620 23:38:34.532449 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:59489
Jun 20, 2026 11:38:35 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 11:38:35 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 11:38:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 11:38:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 11:38:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 11:38:36 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 23:38:36.899158 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.751s user 0.048s sys 0.321s
I20260620 23:38:36.899407 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 23:38:36.899569 388 mini_ranger_kms.cc:401] 127.0.97.20:59489/kms/v1/keys
Jun 20 23:38:36 dist-test-slave-tcgx krb5kdc[12271](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781998716, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 23:38:37 dist-test-slave-tcgx krb5kdc[12271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998716, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 20 23:38:37 dist-test-slave-tcgx krb5kdc[12271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998716, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 20 23:38:40 dist-test-slave-tcgx krb5kdc[12271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998720, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 23:38:40.111153 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42337
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42337
--ranger_config_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:59489/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:38:40.453989 12715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:38:40.454368 12715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:38:40.454530 12715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:38:40.462096 12715 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:38:40.462282 12715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:38:40.462384 12715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:38:40.462486 12715 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:38:40.462566 12715 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:38:40.473165 12715 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:59489/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42337
--ranger_config_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:38:40.474548 12715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:38:40.476106 12715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:38:40.488416 12720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:38:40.488389 12721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:38:40.489678 12723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:38:40.489689 12715 server_base.cc:1034] running on GCE node
I20260620 23:38:40.490600 12715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:38:40.492238 12715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:38:40.493505 12715 hybrid_clock.cc:648] HybridClock initialized: now 1781998720493447 us; error 51 us; skew 500 ppm
I20260620 23:38:40.493917 12715 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:38:40 dist-test-slave-tcgx krb5kdc[12271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998720, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:38:40.500885 12715 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 23:38:40.502959 12715 webserver.cc:460] Webserver started at http://127.0.97.62:34703/ using document root <none> and password file <none>
I20260620 23:38:40.503888 12715 fs_manager.cc:359] Metadata directory not provided
I20260620 23:38:40.504024 12715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:38:40.504383 12715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 23:38:40 dist-test-slave-tcgx krb5kdc[12271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998720, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 23:38:40.569768 12715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/master-0/data/instance:
uuid: "3b3f964a0df440dcbc394fedf4b52796"
format_stamp: "Formatted at 2026-06-20 23:38:40 on dist-test-slave-tcgx"
server_key: "ef0667e48094025ced0cc6108f20d395"
server_key_iv: "29a02624db5b5f939c55266b9dd00f0b"
server_key_version: "kuduclusterkey@0"
I20260620 23:38:40.570621 12715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/master-0/wal/instance:
uuid: "3b3f964a0df440dcbc394fedf4b52796"
format_stamp: "Formatted at 2026-06-20 23:38:40 on dist-test-slave-tcgx"
server_key: "ef0667e48094025ced0cc6108f20d395"
server_key_iv: "29a02624db5b5f939c55266b9dd00f0b"
server_key_version: "kuduclusterkey@0"
I20260620 23:38:40.576228 12715 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 23:38:40.597954 12730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:38:40.599680 12715 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260620 23:38:40.599882 12715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/master-0/wal
uuid: "3b3f964a0df440dcbc394fedf4b52796"
format_stamp: "Formatted at 2026-06-20 23:38:40 on dist-test-slave-tcgx"
server_key: "ef0667e48094025ced0cc6108f20d395"
server_key_iv: "29a02624db5b5f939c55266b9dd00f0b"
server_key_version: "kuduclusterkey@0"
I20260620 23:38:40.600101 12715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:38:40.618070 12715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:38:40.622745 12715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:38:40.623126 12715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:38:40.643333 12715 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42337
I20260620 23:38:40.643352 12781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42337 every 8 connection(s)
I20260620 23:38:40.644888 12715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:38:40.649135 12782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:38:40.649362 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 12715
I20260620 23:38:40.649626 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/master-0/wal/instance
I20260620 23:38:40.650486 388 external_mini_cluster.cc:1428] Setting key c52c4dceaabe2876c726ec3aa50af9bf
I20260620 23:38:40.664440 12782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b3f964a0df440dcbc394fedf4b52796: Bootstrap starting.
I20260620 23:38:40.667737 12782 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b3f964a0df440dcbc394fedf4b52796: Neither blocks nor log segments found. Creating new log.
Jun 20 23:38:40 dist-test-slave-tcgx krb5kdc[12271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998720, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:38:40.668788 12782 log.cc:824] T 00000000000000000000000000000000 P 3b3f964a0df440dcbc394fedf4b52796: Log is configured to *not* fsync() on all Append() calls
I20260620 23:38:40.672390 12782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b3f964a0df440dcbc394fedf4b52796: No bootstrap required, opened a new log
I20260620 23:38:40.679522 12785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:40.653145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60082 (local address 127.0.97.62:42337)
0620 23:38:40.654132 (+ 987us) server_negotiation.cc:206] Beginning negotiation
0620 23:38:40.654150 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 23:38:40.654216 (+ 66us) server_negotiation.cc:407] Connection header received
0620 23:38:40.656515 (+ 2299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:38:40.656548 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:38:40.657138 (+ 590us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:38:40.657642 (+ 504us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:38:40.659218 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:40.661274 (+ 2056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:40.663181 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:40.664238 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:40.664445 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:40.669144 (+ 4699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:38:40.669156 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:38:40.669176 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:38:40.669223 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:38:40.673340 (+ 4117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:38:40.674301 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:38:40.674309 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:38:40.674319 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:38:40.674519 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:38:40.675024 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:38:40.675029 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:38:40.675032 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:38:40.675769 (+ 737us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:38:40.677197 (+ 1428us) server_negotiation.cc:1035] Waiting for connection context
0620 23:38:40.677516 (+ 319us) server_negotiation.cc:299] Negotiation successful
0620 23:38:40.678000 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":724,"thread_start_us":438,"threads_started":1}
I20260620 23:38:40.680516 12782 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b3f964a0df440dcbc394fedf4b52796 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b3f964a0df440dcbc394fedf4b52796" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42337 } }
I20260620 23:38:40.680917 12782 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b3f964a0df440dcbc394fedf4b52796 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:38:40.681108 12782 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b3f964a0df440dcbc394fedf4b52796 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b3f964a0df440dcbc394fedf4b52796, State: Initialized, Role: FOLLOWER
I20260620 23:38:40.681799 12782 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b3f964a0df440dcbc394fedf4b52796 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b3f964a0df440dcbc394fedf4b52796" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42337 } }
I20260620 23:38:40.682307 12782 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b3f964a0df440dcbc394fedf4b52796 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:38:40.682519 12782 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b3f964a0df440dcbc394fedf4b52796 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:38:40.682755 12782 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b3f964a0df440dcbc394fedf4b52796 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:38:40.685110 12782 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b3f964a0df440dcbc394fedf4b52796 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b3f964a0df440dcbc394fedf4b52796" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42337 } }
I20260620 23:38:40.685635 12782 leader_election.cc:302] T 00000000000000000000000000000000 P 3b3f964a0df440dcbc394fedf4b52796 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b3f964a0df440dcbc394fedf4b52796; no voters:
I20260620 23:38:40.686305 12782 leader_election.cc:288] T 00000000000000000000000000000000 P 3b3f964a0df440dcbc394fedf4b52796 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:38:40.686450 12787 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b3f964a0df440dcbc394fedf4b52796 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:38:40.688990 12782 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b3f964a0df440dcbc394fedf4b52796 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:38:40.693516 12787 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b3f964a0df440dcbc394fedf4b52796 [term 1 LEADER]: Becoming Leader. State: Replica: 3b3f964a0df440dcbc394fedf4b52796, State: Running, Role: LEADER
I20260620 23:38:40.694475 12787 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b3f964a0df440dcbc394fedf4b52796 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b3f964a0df440dcbc394fedf4b52796" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42337 } }
I20260620 23:38:40.698283 12788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b3f964a0df440dcbc394fedf4b52796 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b3f964a0df440dcbc394fedf4b52796" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b3f964a0df440dcbc394fedf4b52796" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42337 } } }
I20260620 23:38:40.698770 12788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b3f964a0df440dcbc394fedf4b52796 [sys.catalog]: This master's current role is: LEADER
I20260620 23:38:40.699499 12789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b3f964a0df440dcbc394fedf4b52796 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b3f964a0df440dcbc394fedf4b52796. Latest consensus state: current_term: 1 leader_uuid: "3b3f964a0df440dcbc394fedf4b52796" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b3f964a0df440dcbc394fedf4b52796" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42337 } } }
I20260620 23:38:40.699832 12789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b3f964a0df440dcbc394fedf4b52796 [sys.catalog]: This master's current role is: LEADER
I20260620 23:38:40.704290 12800 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:38:40.714803 12800 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:38:40.723992 12782 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 23:38:40.727608 12800 catalog_manager.cc:1347] Generated new cluster ID: f79f2fa0b9314d2baf15a3bed9f48b75
I20260620 23:38:40.727846 12800 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:38:40.758383 12800 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:38:40.761415 12800 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:38:40.783025 12800 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b3f964a0df440dcbc394fedf4b52796: Generated new TSK 0
I20260620 23:38:40.784348 12800 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 23:38:41.599818 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:59489/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42337
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:38:41 dist-test-slave-tcgx krb5kdc[12271](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781998721, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 23:38:42.066306 12828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:38:42.066763 12828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:38:42.066989 12828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:38:42.078765 12828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:38:42.079007 12828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:38:42.079239 12828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 23:38:42.095343 12828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:59489/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42337
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:38:42.097147 12828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:38:42.099009 12828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:38:42.115127 12838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:38:42.115127 12839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:38:42.116276 12841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:38:42.116772 12828 server_base.cc:1034] running on GCE node
I20260620 23:38:42.117591 12828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:38:42.118884 12828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:38:42.120136 12828 hybrid_clock.cc:648] HybridClock initialized: now 1781998722120069 us; error 68 us; skew 500 ppm
I20260620 23:38:42.120484 12828 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:38:42 dist-test-slave-tcgx krb5kdc[12271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998722, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:38:42.124177 12828 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 23:38:42.125922 12828 webserver.cc:460] Webserver started at http://127.0.97.1:35887/ using document root <none> and password file <none>
I20260620 23:38:42.126740 12828 fs_manager.cc:359] Metadata directory not provided
I20260620 23:38:42.126879 12828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:38:42.127244 12828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 23:38:42 dist-test-slave-tcgx krb5kdc[12271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998722, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 23:38:42.147091 12828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-0/data/instance:
uuid: "b36b856889874cdfaa5ef8af26462d5c"
format_stamp: "Formatted at 2026-06-20 23:38:42 on dist-test-slave-tcgx"
server_key: "20cda849b4d96e58c52dbd08472a54f4"
server_key_iv: "dcb9bde2a64585b884614de444c9d374"
server_key_version: "kuduclusterkey@0"
I20260620 23:38:42.148350 12828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-0/wal/instance:
uuid: "b36b856889874cdfaa5ef8af26462d5c"
format_stamp: "Formatted at 2026-06-20 23:38:42 on dist-test-slave-tcgx"
server_key: "20cda849b4d96e58c52dbd08472a54f4"
server_key_iv: "dcb9bde2a64585b884614de444c9d374"
server_key_version: "kuduclusterkey@0"
I20260620 23:38:42.174824 12828 fs_manager.cc:691] Time spent creating directory manager: real 0.026s user 0.003s sys 0.020s
I20260620 23:38:42.203655 12848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:38:42.208570 12828 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.002s sys 0.007s
I20260620 23:38:42.208914 12828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-0/wal
uuid: "b36b856889874cdfaa5ef8af26462d5c"
format_stamp: "Formatted at 2026-06-20 23:38:42 on dist-test-slave-tcgx"
server_key: "20cda849b4d96e58c52dbd08472a54f4"
server_key_iv: "dcb9bde2a64585b884614de444c9d374"
server_key_version: "kuduclusterkey@0"
I20260620 23:38:42.209327 12828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:38:42.305025 12828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:38:42.311518 12828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:38:42.311914 12828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:38:42.319396 12828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:38:42.327742 12828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:38:42.327913 12828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:38:42.328099 12828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:38:42.328220 12828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 23:38:42 dist-test-slave-tcgx krb5kdc[12271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998721, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 23:38:42.665944 12828 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41993
I20260620 23:38:42.665970 12960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41993 every 8 connection(s)
I20260620 23:38:42.667762 12828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
I20260620 23:38:42.673269 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 12828
I20260620 23:38:42.673507 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-0/wal/instance
I20260620 23:38:42.674417 388 external_mini_cluster.cc:1428] Setting key 0ae782639ef34472ef0797226d007ede
Jun 20 23:38:42 dist-test-slave-tcgx krb5kdc[12271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998722, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260620 23:38:42.709421 12963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:42.671371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42337 (local address 127.0.97.1:52947)
0620 23:38:42.680513 (+ 9142us) negotiation.cc:107] Waiting for socket to connect
0620 23:38:42.680590 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0620 23:38:42.683570 (+ 2980us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:38:42.688396 (+ 4826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:38:42.688410 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:38:42.689186 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:38:42.690175 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:38:42.690205 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:38:42.691816 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:38:42.691822 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:38:42.693973 (+ 2151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:38:42.693991 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:38:42.695161 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:38:42.695168 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:38:42.695474 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:42.696650 (+ 1176us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:38:42.696684 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:38:42.699573 (+ 2889us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:38:42.704201 (+ 4628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:38:42.704214 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:38:42.704235 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:38:42.704939 (+ 704us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:38:42.705456 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:38:42.705461 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:38:42.705464 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:38:42.705730 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:38:42.706448 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:38:42.706457 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:38:42.706986 (+ 529us) client_negotiation.cc:769] Sending connection context
0620 23:38:42.707315 (+ 329us) client_negotiation.cc:240] Negotiation successful
0620 23:38:42.707900 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8724,"thread_start_us":8496,"threads_started":1}
I20260620 23:38:42.711495 12964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:42.672034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52947 (local address 127.0.97.62:42337)
0620 23:38:42.687693 (+ 15659us) server_negotiation.cc:206] Beginning negotiation
0620 23:38:42.687700 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:38:42.687726 (+ 26us) server_negotiation.cc:407] Connection header received
0620 23:38:42.687793 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:38:42.687799 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:38:42.687888 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:38:42.688018 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:38:42.690433 (+ 2415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:42.691625 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:42.694209 (+ 2584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:42.694842 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:42.699829 (+ 4987us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:42.700093 (+ 264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:38:42.700100 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:38:42.700105 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:38:42.700150 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:38:42.704008 (+ 3858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:38:42.705115 (+ 1107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:38:42.705120 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:38:42.705124 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:38:42.705320 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:38:42.705905 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:38:42.705912 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:38:42.705916 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:38:42.706278 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:38:42.710669 (+ 4391us) server_negotiation.cc:1035] Waiting for connection context
0620 23:38:42.710834 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 23:38:42.711121 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15495,"thread_start_us":16102,"threads_started":1}
I20260620 23:38:42.713367 12961 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42337
I20260620 23:38:42.713698 12961 heartbeater.cc:464] Registering TS with master...
I20260620 23:38:42.714587 12961 heartbeater.cc:511] Master 127.0.97.62:42337 requested a full tablet report, sending...
I20260620 23:38:42.717213 12747 ts_manager.cc:194] Registered new tserver with Master: b36b856889874cdfaa5ef8af26462d5c (127.0.97.1:41993)
I20260620 23:38:42.719385 12747 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:52947
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 23:38:42.740403 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:59489/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42337
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:38:42 dist-test-slave-tcgx krb5kdc[12271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998721, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260620 23:38:43.213496 12967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:38:43.213842 12967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:38:43.214013 12967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:38:43.222546 12967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:38:43.222678 12967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:38:43.222854 12967 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 23:38:43.233124 12967 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:59489/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42337
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:38:43.234843 12967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:38:43.236842 12967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:38:43.252254 12984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:38:43.252357 12982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:38:43.252537 12981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:38:43.256799 12967 server_base.cc:1034] running on GCE node
I20260620 23:38:43.257535 12967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:38:43.258607 12967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:38:43.259850 12967 hybrid_clock.cc:648] HybridClock initialized: now 1781998723259769 us; error 80 us; skew 500 ppm
I20260620 23:38:43.260193 12967 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:38:43 dist-test-slave-tcgx krb5kdc[12271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998723, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:38:43.263855 12967 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 23:38:43.265547 12967 webserver.cc:460] Webserver started at http://127.0.97.2:43001/ using document root <none> and password file <none>
I20260620 23:38:43.266306 12967 fs_manager.cc:359] Metadata directory not provided
I20260620 23:38:43.266430 12967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:38:43.266784 12967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 23:38:43 dist-test-slave-tcgx krb5kdc[12271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998723, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 23:38:43.284890 12967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-1/data/instance:
uuid: "deed2cf1304549448c299f8d37361198"
format_stamp: "Formatted at 2026-06-20 23:38:43 on dist-test-slave-tcgx"
server_key: "09c49486569b7c882b18852fcea5de67"
server_key_iv: "7c48947848f08a209b21f85c32cf6055"
server_key_version: "kuduclusterkey@0"
I20260620 23:38:43.286638 12967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-1/wal/instance:
uuid: "deed2cf1304549448c299f8d37361198"
format_stamp: "Formatted at 2026-06-20 23:38:43 on dist-test-slave-tcgx"
server_key: "09c49486569b7c882b18852fcea5de67"
server_key_iv: "7c48947848f08a209b21f85c32cf6055"
server_key_version: "kuduclusterkey@0"
I20260620 23:38:43.294356 12967 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.000s sys 0.006s
I20260620 23:38:43.313714 12991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:38:43.315626 12967 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260620 23:38:43.315878 12967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-1/wal
uuid: "deed2cf1304549448c299f8d37361198"
format_stamp: "Formatted at 2026-06-20 23:38:43 on dist-test-slave-tcgx"
server_key: "09c49486569b7c882b18852fcea5de67"
server_key_iv: "7c48947848f08a209b21f85c32cf6055"
server_key_version: "kuduclusterkey@0"
I20260620 23:38:43.316118 12967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:38:43.345736 12967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:38:43.350081 12967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:38:43.350445 12967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:38:43.351604 12967 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:38:43.353267 12967 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:38:43.353370 12967 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:38:43.353497 12967 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:38:43.353592 12967 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:38:43.400755 12967 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33317
I20260620 23:38:43.400755 13103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33317 every 8 connection(s)
I20260620 23:38:43.402189 12967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
I20260620 23:38:43.404063 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 12967
I20260620 23:38:43.404297 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-1/wal/instance
I20260620 23:38:43.404989 388 external_mini_cluster.cc:1428] Setting key 23eebeac7cb156a20132af05e48ff44d
Jun 20 23:38:43 dist-test-slave-tcgx krb5kdc[12271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998723, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 23:38:43.450618 13106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:43.418487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40969 (local address 127.0.97.62:42337)
0620 23:38:43.429898 (+ 11411us) server_negotiation.cc:206] Beginning negotiation
0620 23:38:43.429903 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:38:43.429921 (+ 18us) server_negotiation.cc:407] Connection header received
0620 23:38:43.429974 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:38:43.429978 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:38:43.430048 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:38:43.430187 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:38:43.432283 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:43.433440 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:43.437493 (+ 4053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:43.438168 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:43.438307 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:43.442836 (+ 4529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:38:43.442843 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:38:43.442848 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:38:43.442891 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:38:43.446524 (+ 3633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:38:43.447604 (+ 1080us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:38:43.447610 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:38:43.447614 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:38:43.447791 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:38:43.448348 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:38:43.448354 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:38:43.448357 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:38:43.448748 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:38:43.448891 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 23:38:43.449957 (+ 1066us) server_negotiation.cc:299] Negotiation successful
0620 23:38:43.450222 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11287,"thread_start_us":11812,"threads_started":1}
I20260620 23:38:43.451815 13107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:43.419527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42337 (local address 127.0.97.2:40969)
0620 23:38:43.425703 (+ 6176us) negotiation.cc:107] Waiting for socket to connect
0620 23:38:43.425786 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0620 23:38:43.429267 (+ 3481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:38:43.430493 (+ 1226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:38:43.430507 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:38:43.431150 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:38:43.432022 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:38:43.432049 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:38:43.435100 (+ 3051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:38:43.435106 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:38:43.437246 (+ 2140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:38:43.437264 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:38:43.438622 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:38:43.438628 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:38:43.439008 (+ 380us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:43.439862 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:38:43.439892 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:38:43.442527 (+ 2635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:38:43.446776 (+ 4249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:38:43.446786 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:38:43.446804 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:38:43.447434 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:38:43.447952 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:38:43.447957 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:38:43.447961 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:38:43.448210 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:38:43.448975 (+ 765us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:38:43.448987 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:38:43.449454 (+ 467us) client_negotiation.cc:769] Sending connection context
0620 23:38:43.449800 (+ 346us) client_negotiation.cc:240] Negotiation successful
0620 23:38:43.450350 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5728,"thread_start_us":6855,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260620 23:38:43.453433 13104 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42337
I20260620 23:38:43.453760 13104 heartbeater.cc:464] Registering TS with master...
I20260620 23:38:43.454572 13104 heartbeater.cc:511] Master 127.0.97.62:42337 requested a full tablet report, sending...
I20260620 23:38:43.456209 12747 ts_manager.cc:194] Registered new tserver with Master: deed2cf1304549448c299f8d37361198 (127.0.97.2:33317)
I20260620 23:38:43.457301 12747 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:40969
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 23:38:43.487433 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:59489/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42337
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:38:43.723220 12961 heartbeater.cc:503] Master 127.0.97.62:42337 was elected leader, sending a full tablet report...
W20260620 23:38:43.831840 13110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:38:43.832139 13110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:38:43.832300 13110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:38:43.839612 13110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:38:43.839725 13110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:38:43.839910 13110 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 23:38:43.849931 13110 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:59489/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42337
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:38:43.851378 13110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:38:43.852682 13110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:38:43.865770 13116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:38:43.865886 13115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:38:43.865865 13118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:38:43.866645 13110 server_base.cc:1034] running on GCE node
I20260620 23:38:43.867369 13110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:38:43.868412 13110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:38:43.869609 13110 hybrid_clock.cc:648] HybridClock initialized: now 1781998723869576 us; error 34 us; skew 500 ppm
I20260620 23:38:43.869894 13110 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:38:43 dist-test-slave-tcgx krb5kdc[12271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998723, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:38:43.873127 13110 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 23:38:43.874593 13110 webserver.cc:460] Webserver started at http://127.0.97.3:39535/ using document root <none> and password file <none>
I20260620 23:38:43.875380 13110 fs_manager.cc:359] Metadata directory not provided
I20260620 23:38:43.875494 13110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:38:43.875767 13110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 23:38:43 dist-test-slave-tcgx krb5kdc[12271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998723, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 23:38:43.892622 13110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-2/data/instance:
uuid: "e87feaa0e06d4a858c302e28cd3b0afc"
format_stamp: "Formatted at 2026-06-20 23:38:43 on dist-test-slave-tcgx"
server_key: "764e5944c95e353d0b7182b9a38f9bc6"
server_key_iv: "ab43a6a4b56624b4205b0bf0d23940c5"
server_key_version: "kuduclusterkey@0"
I20260620 23:38:43.893520 13110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-2/wal/instance:
uuid: "e87feaa0e06d4a858c302e28cd3b0afc"
format_stamp: "Formatted at 2026-06-20 23:38:43 on dist-test-slave-tcgx"
server_key: "764e5944c95e353d0b7182b9a38f9bc6"
server_key_iv: "ab43a6a4b56624b4205b0bf0d23940c5"
server_key_version: "kuduclusterkey@0"
I20260620 23:38:43.899740 13110 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260620 23:38:43.917555 13125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:38:43.919356 13110 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260620 23:38:43.919540 13110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-2/wal
uuid: "e87feaa0e06d4a858c302e28cd3b0afc"
format_stamp: "Formatted at 2026-06-20 23:38:43 on dist-test-slave-tcgx"
server_key: "764e5944c95e353d0b7182b9a38f9bc6"
server_key_iv: "ab43a6a4b56624b4205b0bf0d23940c5"
server_key_version: "kuduclusterkey@0"
I20260620 23:38:43.919793 13110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:38:43.937350 13110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:38:43.941635 13110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:38:43.941949 13110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:38:43.943095 13110 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:38:43.944563 13110 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:38:43.944672 13110 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:38:43.944830 13110 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:38:43.944901 13110 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:38:43.983467 13110 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38123
I20260620 23:38:43.983492 13237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38123 every 8 connection(s)
I20260620 23:38:43.984925 13110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
I20260620 23:38:43.991747 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 13110
I20260620 23:38:43.991961 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-2/wal/instance
I20260620 23:38:43.992648 388 external_mini_cluster.cc:1428] Setting key 5c64736ee3741f17215ba89389a5b1ec
Jun 20 23:38:43 dist-test-slave-tcgx krb5kdc[12271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998723, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:38:44.009362 13240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:43.987810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35217 (local address 127.0.97.62:42337)
0620 23:38:43.988338 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0620 23:38:43.988344 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:38:43.990335 (+ 1991us) server_negotiation.cc:407] Connection header received
0620 23:38:43.991419 (+ 1084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:38:43.991426 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:38:43.991538 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:38:43.991663 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:38:43.993678 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:43.995085 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:43.997202 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:43.997879 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:43.998011 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:44.001426 (+ 3415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:38:44.001433 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:38:44.001438 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:38:44.001491 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:38:44.005371 (+ 3880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:38:44.006291 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:38:44.006297 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:38:44.006301 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:38:44.006475 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:38:44.007166 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:38:44.007173 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:38:44.007177 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:38:44.007628 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:38:44.007811 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 23:38:44.008751 (+ 940us) server_negotiation.cc:299] Negotiation successful
0620 23:38:44.008999 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":269,"threads_started":1}
I20260620 23:38:44.010394 13241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:43.988413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42337 (local address 127.0.97.3:35217)
0620 23:38:43.989224 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0620 23:38:43.989272 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 23:38:43.991100 (+ 1828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:38:43.991970 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:38:43.991985 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:38:43.992603 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:38:43.993447 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:38:43.993468 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:38:43.995250 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:38:43.995255 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:38:43.997044 (+ 1789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:38:43.997056 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:38:43.998101 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:38:43.998106 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:38:43.998322 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:43.999073 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:38:43.999096 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:38:44.001172 (+ 2076us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:38:44.005591 (+ 4419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:38:44.005600 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:38:44.005621 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:38:44.006144 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:38:44.006663 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:38:44.006669 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:38:44.006673 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:38:44.007017 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:38:44.007811 (+ 794us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:38:44.007821 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:38:44.008258 (+ 437us) client_negotiation.cc:769] Sending connection context
0620 23:38:44.008642 (+ 384us) client_negotiation.cc:240] Negotiation successful
0620 23:38:44.009207 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":361,"threads_started":1}
I20260620 23:38:44.011816 13238 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42337
I20260620 23:38:44.012163 13238 heartbeater.cc:464] Registering TS with master...
I20260620 23:38:44.013042 13238 heartbeater.cc:511] Master 127.0.97.62:42337 requested a full tablet report, sending...
I20260620 23:38:44.014684 12747 ts_manager.cc:194] Registered new tserver with Master: e87feaa0e06d4a858c302e28cd3b0afc (127.0.97.3:38123)
I20260620 23:38:44.015902 12747 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:35217
I20260620 23:38:44.022027 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:38:44.023845 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 12828
Jun 20 23:38:44 dist-test-slave-tcgx krb5kdc[12271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998720, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 23:38:44.041146 13243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:44.025050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52868 (local address 127.0.97.1:41993)
0620 23:38:44.025677 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0620 23:38:44.025684 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:38:44.025709 (+ 25us) server_negotiation.cc:407] Connection header received
0620 23:38:44.025765 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:38:44.025773 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:38:44.025981 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:38:44.026106 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:38:44.027698 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:44.028723 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:44.030192 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:44.030750 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:44.030894 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:44.035509 (+ 4615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:38:44.035521 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:38:44.035536 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:38:44.035577 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:38:44.038198 (+ 2621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:38:44.038901 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:38:44.038909 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:38:44.038911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:38:44.039104 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:38:44.039494 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:38:44.039500 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:38:44.039504 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:38:44.039871 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:38:44.040194 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0620 23:38:44.040501 (+ 307us) server_negotiation.cc:299] Negotiation successful
0620 23:38:44.040780 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":314,"threads_started":1}
I20260620 23:38:44.042479 12956 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:38:44.127059 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 12828
I20260620 23:38:44.143083 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 12967
Jun 20 23:38:44 dist-test-slave-tcgx krb5kdc[12271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998720, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 23:38:44.159335 13245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:44.143976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55922 (local address 127.0.97.2:33317)
0620 23:38:44.144536 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0620 23:38:44.144544 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:38:44.144570 (+ 26us) server_negotiation.cc:407] Connection header received
0620 23:38:44.144648 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:38:44.144654 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:38:44.144874 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:38:44.145006 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:38:44.146190 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:44.147261 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:44.148726 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:44.149302 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:44.149403 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:44.153646 (+ 4243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:38:44.153657 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:38:44.153671 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:38:44.153701 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:38:44.156640 (+ 2939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:38:44.157439 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:38:44.157446 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:38:44.157448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:38:44.157580 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:38:44.157891 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:38:44.157894 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:38:44.157896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:38:44.158289 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:38:44.158513 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 23:38:44.158779 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 23:38:44.159043 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":303,"threads_started":1}
I20260620 23:38:44.160631 13099 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:38:44.243678 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 12967
I20260620 23:38:44.259707 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 13110
Jun 20 23:38:44 dist-test-slave-tcgx krb5kdc[12271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998720, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 23:38:44.276787 13247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:44.260905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58662 (local address 127.0.97.3:38123)
0620 23:38:44.261538 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0620 23:38:44.261545 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:38:44.261574 (+ 29us) server_negotiation.cc:407] Connection header received
0620 23:38:44.261635 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:38:44.261641 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:38:44.261867 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:38:44.262044 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:38:44.263425 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:44.264442 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:44.265782 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:44.266358 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:44.266507 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:44.271056 (+ 4549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:38:44.271071 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:38:44.271087 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:38:44.271123 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:38:44.273792 (+ 2669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:38:44.274524 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:38:44.274541 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:38:44.274543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:38:44.274675 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:38:44.275113 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:38:44.275119 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:38:44.275123 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:38:44.275481 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:38:44.275817 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0620 23:38:44.276148 (+ 331us) server_negotiation.cc:299] Negotiation successful
0620 23:38:44.276434 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":340,"threads_started":1}
I20260620 23:38:44.278211 13233 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:38:44.367528 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 13110
I20260620 23:38:44.383687 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 12715
I20260620 23:38:44.385077 12777 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:38:44.491611 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 12715
I20260620 23:38:44.516333 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42337
--webserver_interface=127.0.97.62
--webserver_port=34703
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42337
--ranger_config_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:59489/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:59489/kms with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:38:44.865463 13250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:38:44.865783 13250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:38:44.865945 13250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:38:44.873502 13250 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:38:44.873621 13250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:38:44.873708 13250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:38:44.873805 13250 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:38:44.873893 13250 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:38:44.884533 13250 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:59489/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42337
--ranger_config_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=34703
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:38:44.885982 13250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:38:44.887476 13250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:38:44.899721 13255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:38:44.899847 13256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:38:44.899873 13258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:38:44.901319 13250 server_base.cc:1034] running on GCE node
I20260620 23:38:44.902171 13250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:38:44.903856 13250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:38:44.905154 13250 hybrid_clock.cc:648] HybridClock initialized: now 1781998724905104 us; error 49 us; skew 500 ppm
I20260620 23:38:44.905606 13250 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:38:44 dist-test-slave-tcgx krb5kdc[12271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998724, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:38:44.909047 13250 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 23:38:44.910631 13250 webserver.cc:460] Webserver started at http://127.0.97.62:34703/ using document root <none> and password file <none>
I20260620 23:38:44.911386 13250 fs_manager.cc:359] Metadata directory not provided
I20260620 23:38:44.911537 13250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 23:38:44 dist-test-slave-tcgx krb5kdc[12271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998724, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 23:38:44.965219 13250 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 23:38:44.969228 13266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:38:44.971094 13250 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 23:38:44.971315 13250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/master-0/wal
uuid: "3b3f964a0df440dcbc394fedf4b52796"
format_stamp: "Formatted at 2026-06-20 23:38:40 on dist-test-slave-tcgx"
server_key: "ef0667e48094025ced0cc6108f20d395"
server_key_iv: "29a02624db5b5f939c55266b9dd00f0b"
server_key_version: "kuduclusterkey@0"
I20260620 23:38:44.972726 13250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:38:45.000641 13250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:38:45.004629 13250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:38:45.004905 13250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:38:45.024242 13250 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42337
I20260620 23:38:45.024261 13317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42337 every 8 connection(s)
I20260620 23:38:45.025833 13250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:38:45.030440 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 13250
I20260620 23:38:45.031394 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:59489/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:41993
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:42337
--webserver_port=35887
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:59489/kms with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:38:45.033157 13318 sys_catalog.cc:263] Verifying existing consensus state
I20260620 23:38:45.036568 13318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b3f964a0df440dcbc394fedf4b52796: Bootstrap starting.
I20260620 23:38:45.056521 13318 log.cc:824] T 00000000000000000000000000000000 P 3b3f964a0df440dcbc394fedf4b52796: Log is configured to *not* fsync() on all Append() calls
I20260620 23:38:45.065593 13318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b3f964a0df440dcbc394fedf4b52796: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:38:45.066148 13318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b3f964a0df440dcbc394fedf4b52796: Bootstrap complete.
I20260620 23:38:45.075306 13318 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b3f964a0df440dcbc394fedf4b52796 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b3f964a0df440dcbc394fedf4b52796" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42337 } }
I20260620 23:38:45.076215 13318 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b3f964a0df440dcbc394fedf4b52796 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b3f964a0df440dcbc394fedf4b52796, State: Initialized, Role: FOLLOWER
I20260620 23:38:45.076920 13318 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b3f964a0df440dcbc394fedf4b52796 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b3f964a0df440dcbc394fedf4b52796" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42337 } }
I20260620 23:38:45.077162 13318 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b3f964a0df440dcbc394fedf4b52796 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:38:45.077292 13318 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b3f964a0df440dcbc394fedf4b52796 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:38:45.077483 13318 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b3f964a0df440dcbc394fedf4b52796 [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:38:45.081485 13318 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b3f964a0df440dcbc394fedf4b52796 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b3f964a0df440dcbc394fedf4b52796" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42337 } }
I20260620 23:38:45.082021 13318 leader_election.cc:302] T 00000000000000000000000000000000 P 3b3f964a0df440dcbc394fedf4b52796 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b3f964a0df440dcbc394fedf4b52796; no voters:
I20260620 23:38:45.082667 13318 leader_election.cc:288] T 00000000000000000000000000000000 P 3b3f964a0df440dcbc394fedf4b52796 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 23:38:45.082813 13322 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b3f964a0df440dcbc394fedf4b52796 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 23:38:45.084031 13322 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b3f964a0df440dcbc394fedf4b52796 [term 2 LEADER]: Becoming Leader. State: Replica: 3b3f964a0df440dcbc394fedf4b52796, State: Running, Role: LEADER
I20260620 23:38:45.084658 13322 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b3f964a0df440dcbc394fedf4b52796 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b3f964a0df440dcbc394fedf4b52796" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42337 } }
I20260620 23:38:45.085084 13318 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b3f964a0df440dcbc394fedf4b52796 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:38:45.087479 13324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b3f964a0df440dcbc394fedf4b52796 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b3f964a0df440dcbc394fedf4b52796. Latest consensus state: current_term: 2 leader_uuid: "3b3f964a0df440dcbc394fedf4b52796" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b3f964a0df440dcbc394fedf4b52796" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42337 } } }
I20260620 23:38:45.087927 13324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b3f964a0df440dcbc394fedf4b52796 [sys.catalog]: This master's current role is: LEADER
I20260620 23:38:45.088526 13323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b3f964a0df440dcbc394fedf4b52796 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3b3f964a0df440dcbc394fedf4b52796" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b3f964a0df440dcbc394fedf4b52796" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42337 } } }
I20260620 23:38:45.088785 13328 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:38:45.088820 13323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b3f964a0df440dcbc394fedf4b52796 [sys.catalog]: This master's current role is: LEADER
I20260620 23:38:45.098891 13328 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:38:45.100931 13328 catalog_manager.cc:1259] Loaded cluster ID: f79f2fa0b9314d2baf15a3bed9f48b75
I20260620 23:38:45.101114 13328 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:38:45.106503 13328 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:38:45.108696 13328 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3b3f964a0df440dcbc394fedf4b52796: Loaded TSK: 0
I20260620 23:38:45.110916 13328 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:38:45.115967 13318 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260620 23:38:45.474987 13320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:38:45.475509 13320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:38:45.475723 13320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:38:45.487537 13320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:38:45.487787 13320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:38:45.488034 13320 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 23:38:45.504101 13320 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:59489/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:41993
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=35887
--tserver_master_addrs=127.0.97.62:42337
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:38:45.506206 13320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:38:45.508257 13320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:38:45.552695 13357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:38:45.555159 13356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:38:45.560200 13320 server_base.cc:1034] running on GCE node
W20260620 23:38:45.560753 13359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:38:45.561704 13320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:38:45.562881 13320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:38:45.564175 13320 hybrid_clock.cc:648] HybridClock initialized: now 1781998725564107 us; error 64 us; skew 500 ppm
I20260620 23:38:45.564538 13320 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:38:45 dist-test-slave-tcgx krb5kdc[12271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:38:45.573249 13320 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 23:38:45.575585 13320 webserver.cc:460] Webserver started at http://127.0.97.1:35887/ using document root <none> and password file <none>
I20260620 23:38:45.576511 13320 fs_manager.cc:359] Metadata directory not provided
I20260620 23:38:45.576717 13320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 23:38:45 dist-test-slave-tcgx krb5kdc[12271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 23:38:45.631160 13320 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:38:45.638609 13367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:38:45.640453 13320 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.003s sys 0.000s
I20260620 23:38:45.640661 13320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-0/wal
uuid: "b36b856889874cdfaa5ef8af26462d5c"
format_stamp: "Formatted at 2026-06-20 23:38:42 on dist-test-slave-tcgx"
server_key: "20cda849b4d96e58c52dbd08472a54f4"
server_key_iv: "dcb9bde2a64585b884614de444c9d374"
server_key_version: "kuduclusterkey@0"
I20260620 23:38:45.641734 13320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:38:45.675487 13320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:38:45.682652 13320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:38:45.683068 13320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:38:45.690791 13320 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:38:45.700614 13320 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:38:45.700846 13320 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 23:38:45.701004 13320 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:38:45.701128 13320 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:38:45.768419 13320 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41993
I20260620 23:38:45.768461 13480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41993 every 8 connection(s)
I20260620 23:38:45.770330 13320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
I20260620 23:38:45.774987 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 13320
I20260620 23:38:45.775808 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:59489/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:33317
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:42337
--webserver_port=43001
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:59489/kms with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:38:45 dist-test-slave-tcgx krb5kdc[12271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:38:45.815215 13484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:45.776395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42337 (local address 127.0.97.1:48197)
0620 23:38:45.785252 (+ 8857us) negotiation.cc:107] Waiting for socket to connect
0620 23:38:45.785318 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0620 23:38:45.790647 (+ 5329us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:38:45.792806 (+ 2159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:38:45.792820 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:38:45.793500 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:38:45.794337 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:38:45.794367 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:38:45.796915 (+ 2548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:38:45.796922 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:38:45.799058 (+ 2136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:38:45.799076 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:38:45.800950 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:38:45.800960 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:38:45.801249 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:45.802062 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:38:45.802101 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:38:45.805069 (+ 2968us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:38:45.809686 (+ 4617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:38:45.809695 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:38:45.809714 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:38:45.810343 (+ 629us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:38:45.810920 (+ 577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:38:45.810925 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:38:45.810928 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:38:45.811190 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:38:45.812290 (+ 1100us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:38:45.812297 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:38:45.812721 (+ 424us) client_negotiation.cc:769] Sending connection context
0620 23:38:45.813036 (+ 315us) client_negotiation.cc:240] Negotiation successful
0620 23:38:45.813623 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8433,"thread_start_us":10855,"threads_started":1}
I20260620 23:38:45.815738 13483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:45.776240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48197 (local address 127.0.97.62:42337)
0620 23:38:45.787491 (+ 11251us) server_negotiation.cc:206] Beginning negotiation
0620 23:38:45.787512 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 23:38:45.787573 (+ 61us) server_negotiation.cc:407] Connection header received
0620 23:38:45.791513 (+ 3940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:38:45.791538 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:38:45.792142 (+ 604us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:38:45.792610 (+ 468us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:38:45.794600 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:45.796660 (+ 2060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:45.799257 (+ 2597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:45.800324 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:45.800515 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:45.805419 (+ 4904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:38:45.805459 (+ 40us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:38:45.805483 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:38:45.805552 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:38:45.809487 (+ 3935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:38:45.810540 (+ 1053us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:38:45.810549 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:38:45.810558 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:38:45.810784 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:38:45.811335 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:38:45.811339 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:38:45.811342 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:38:45.812008 (+ 666us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:38:45.812213 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 23:38:45.814105 (+ 1892us) server_negotiation.cc:299] Negotiation successful
0620 23:38:45.814513 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10982,"thread_start_us":10738,"threads_started":1}
I20260620 23:38:45.819027 13481 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42337
I20260620 23:38:45.819377 13481 heartbeater.cc:464] Registering TS with master...
I20260620 23:38:45.820319 13481 heartbeater.cc:511] Master 127.0.97.62:42337 requested a full tablet report, sending...
W20260620 23:38:45.823597 13481 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:42337 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 23:38:45.823851 13481 heartbeater.cc:464] Registering TS with master...
I20260620 23:38:45.824154 13481 heartbeater.cc:511] Master 127.0.97.62:42337 requested a full tablet report, sending...
I20260620 23:38:45.825415 13481 heartbeater.cc:464] Registering TS with master...
I20260620 23:38:45.825774 13481 heartbeater.cc:511] Master 127.0.97.62:42337 requested a full tablet report, sending...
W20260620 23:38:45.827001 13481 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 23:38:46.170275 13485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:38:46.170603 13485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:38:46.170781 13485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:38:46.179311 13485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:38:46.179462 13485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:38:46.179636 13485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 23:38:46.190230 13485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:59489/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:33317
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=43001
--tserver_master_addrs=127.0.97.62:42337
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:38:46.191668 13485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:38:46.193032 13485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:38:46.210567 13500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:38:46.211982 13498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:38:46.212119 13502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:38:46.215974 13485 server_base.cc:1034] running on GCE node
I20260620 23:38:46.216713 13485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:38:46.217796 13485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:38:46.219055 13485 hybrid_clock.cc:648] HybridClock initialized: now 1781998726218988 us; error 59 us; skew 500 ppm
I20260620 23:38:46.219470 13485 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:38:46 dist-test-slave-tcgx krb5kdc[12271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998726, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:38:46.222836 13485 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 23:38:46.228391 13485 webserver.cc:460] Webserver started at http://127.0.97.2:43001/ using document root <none> and password file <none>
I20260620 23:38:46.229189 13485 fs_manager.cc:359] Metadata directory not provided
I20260620 23:38:46.229360 13485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 23:38:46 dist-test-slave-tcgx krb5kdc[12271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998726, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 23:38:46.256924 13485 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 23:38:46.260986 13510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:38:46.262907 13485 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 23:38:46.263136 13485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-1/wal
uuid: "deed2cf1304549448c299f8d37361198"
format_stamp: "Formatted at 2026-06-20 23:38:43 on dist-test-slave-tcgx"
server_key: "09c49486569b7c882b18852fcea5de67"
server_key_iv: "7c48947848f08a209b21f85c32cf6055"
server_key_version: "kuduclusterkey@0"
I20260620 23:38:46.264628 13485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:38:46.322512 13485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:38:46.328377 13485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:38:46.328789 13485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:38:46.336249 13485 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:38:46.344537 13485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:38:46.344674 13485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:38:46.344854 13485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:38:46.344941 13485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 23:38:46 dist-test-slave-tcgx krb5kdc[12271](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781998726, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:38:46.403438 13485 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33317
I20260620 23:38:46.403481 13624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33317 every 8 connection(s)
I20260620 23:38:46.405066 13485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
I20260620 23:38:46.411777 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 13485
I20260620 23:38:46.412573 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:59489/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:38123
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:42337
--webserver_port=39535
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:59489/kms with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:38:46 dist-test-slave-tcgx krb5kdc[12271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998726, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:38:46.461506 13628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:46.410124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39275 (local address 127.0.97.62:42337)
0620 23:38:46.435528 (+ 25404us) server_negotiation.cc:206] Beginning negotiation
0620 23:38:46.435537 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:38:46.440191 (+ 4654us) server_negotiation.cc:407] Connection header received
0620 23:38:46.440286 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:38:46.440295 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:38:46.440417 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:38:46.440544 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:38:46.444135 (+ 3591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:46.445357 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:46.448071 (+ 2714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:46.448759 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:46.448859 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:46.453415 (+ 4556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:38:46.453423 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:38:46.453429 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:38:46.453474 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:38:46.457533 (+ 4059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:38:46.458619 (+ 1086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:38:46.458625 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:38:46.458628 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:38:46.458811 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:38:46.459411 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:38:46.459416 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:38:46.459419 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:38:46.459805 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:38:46.459961 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 23:38:46.460912 (+ 951us) server_negotiation.cc:299] Negotiation successful
0620 23:38:46.461150 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25203,"thread_start_us":25415,"threads_started":1}
I20260620 23:38:46.464040 13627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:46.409298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42337 (local address 127.0.97.2:39275)
0620 23:38:46.436350 (+ 27052us) negotiation.cc:107] Waiting for socket to connect
0620 23:38:46.436428 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0620 23:38:46.439565 (+ 3137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:38:46.442056 (+ 2491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:38:46.442073 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:38:46.442852 (+ 779us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:38:46.443866 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:38:46.443896 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:38:46.445586 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:38:46.445592 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:38:46.447868 (+ 2276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:38:46.447899 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:38:46.449242 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:38:46.449251 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:38:46.449556 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:46.450388 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:38:46.450420 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:38:46.453125 (+ 2705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:38:46.457784 (+ 4659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:38:46.457795 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:38:46.457815 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:38:46.458462 (+ 647us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:38:46.458995 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:38:46.459000 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:38:46.459003 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:38:46.459268 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:38:46.460064 (+ 796us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:38:46.460074 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:38:46.460469 (+ 395us) client_negotiation.cc:769] Sending connection context
0620 23:38:46.462155 (+ 1686us) client_negotiation.cc:240] Negotiation successful
0620 23:38:46.462732 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26631,"thread_start_us":31285,"threads_started":1}
I20260620 23:38:46.466197 13625 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42337
I20260620 23:38:46.466519 13625 heartbeater.cc:464] Registering TS with master...
I20260620 23:38:46.467489 13625 heartbeater.cc:511] Master 127.0.97.62:42337 requested a full tablet report, sending...
I20260620 23:38:46.469832 13283 ts_manager.cc:194] Registered new tserver with Master: deed2cf1304549448c299f8d37361198 (127.0.97.2:33317)
I20260620 23:38:46.472095 13283 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:39275
I20260620 23:38:46.827126 13481 heartbeater.cc:464] Registering TS with master...
I20260620 23:38:46.827580 13481 heartbeater.cc:511] Master 127.0.97.62:42337 requested a full tablet report, sending...
I20260620 23:38:46.828788 13283 ts_manager.cc:194] Registered new tserver with Master: b36b856889874cdfaa5ef8af26462d5c (127.0.97.1:41993)
I20260620 23:38:46.829869 13283 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:48197
Jun 20 23:38:46 dist-test-slave-tcgx krb5kdc[12271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998726, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260620 23:38:46.884742 13629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:38:46.885228 13629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:38:46.885416 13629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:38:46.899595 13629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:38:46.899780 13629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:38:46.899989 13629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 23:38:46.915242 13629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:59489/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:38123
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=39535
--tserver_master_addrs=127.0.97.62:42337
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:38:46.917063 13629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:38:46.918817 13629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:38:46.942227 13635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:38:46.942757 13636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:38:46.951570 13638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:38:46.952009 13629 server_base.cc:1034] running on GCE node
I20260620 23:38:46.952754 13629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:38:46.953877 13629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:38:46.955090 13629 hybrid_clock.cc:648] HybridClock initialized: now 1781998726955021 us; error 59 us; skew 500 ppm
I20260620 23:38:46.955447 13629 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:38:46 dist-test-slave-tcgx krb5kdc[12271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998726, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:38:46.958817 13629 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 23:38:46.961084 13629 webserver.cc:460] Webserver started at http://127.0.97.3:39535/ using document root <none> and password file <none>
I20260620 23:38:46.961845 13629 fs_manager.cc:359] Metadata directory not provided
I20260620 23:38:46.962013 13629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 23:38:46 dist-test-slave-tcgx krb5kdc[12271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998726, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 23:38:46.994518 13629 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 23:38:46.998095 13646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:38:46.999855 13629 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.002s
I20260620 23:38:47.000052 13629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-2/wal
uuid: "e87feaa0e06d4a858c302e28cd3b0afc"
format_stamp: "Formatted at 2026-06-20 23:38:43 on dist-test-slave-tcgx"
server_key: "764e5944c95e353d0b7182b9a38f9bc6"
server_key_iv: "ab43a6a4b56624b4205b0bf0d23940c5"
server_key_version: "kuduclusterkey@0"
I20260620 23:38:47.001134 13629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:38:47.077710 13629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:38:47.083546 13629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:38:47.083900 13629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:38:47.085258 13629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:38:47.087208 13629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:38:47.087329 13629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:38:47.087448 13629 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:38:47.087534 13629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:38:47.158835 13629 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38123
I20260620 23:38:47.158869 13758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38123 every 8 connection(s)
I20260620 23:38:47.160885 13629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
I20260620 23:38:47.161283 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 13629
I20260620 23:38:47.193722 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:47.163018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58246 (local address 127.0.97.62:42337)
0620 23:38:47.164993 (+ 1975us) server_negotiation.cc:206] Beginning negotiation
0620 23:38:47.165000 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:38:47.166992 (+ 1992us) server_negotiation.cc:407] Connection header received
0620 23:38:47.167073 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:38:47.167079 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:38:47.167179 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:38:47.167283 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:38:47.170353 (+ 3070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:47.171487 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:47.173618 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:47.174334 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:47.174461 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:47.179770 (+ 5309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:38:47.179776 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:38:47.179780 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:38:47.179830 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:38:47.182887 (+ 3057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:38:47.186853 (+ 3966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:38:47.186861 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:38:47.186865 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:38:47.187105 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:38:47.191725 (+ 4620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:38:47.191731 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:38:47.191736 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:38:47.192129 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:38:47.192401 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0620 23:38:47.193044 (+ 643us) server_negotiation.cc:299] Negotiation successful
0620 23:38:47.193320 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1850,"thread_start_us":1744,"threads_started":1}
Jun 20 23:38:47 dist-test-slave-tcgx krb5kdc[12271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998726, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:38:47.213997 13764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:47.173959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54715 (local address 127.0.97.62:42337)
0620 23:38:47.179502 (+ 5543us) server_negotiation.cc:206] Beginning negotiation
0620 23:38:47.179508 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:38:47.186835 (+ 7327us) server_negotiation.cc:407] Connection header received
0620 23:38:47.190833 (+ 3998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:38:47.190840 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:38:47.191000 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:38:47.191120 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:38:47.196200 (+ 5080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:47.197421 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:47.199972 (+ 2551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:47.200636 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:47.200751 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:47.205632 (+ 4881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:38:47.205641 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:38:47.205646 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:38:47.205701 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:38:47.209800 (+ 4099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:38:47.210867 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:38:47.210873 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:38:47.210877 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:38:47.211062 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:38:47.211689 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:38:47.211695 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:38:47.211699 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:38:47.212076 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:38:47.212217 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 23:38:47.213318 (+ 1101us) server_negotiation.cc:299] Negotiation successful
0620 23:38:47.213590 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5411,"thread_start_us":5919,"threads_started":1}
I20260620 23:38:47.215687 13763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:47.173356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42337 (local address 127.0.97.3:54715)
0620 23:38:47.186536 (+ 13180us) negotiation.cc:107] Waiting for socket to connect
0620 23:38:47.186633 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0620 23:38:47.190357 (+ 3724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:38:47.194301 (+ 3944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:38:47.194315 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:38:47.195037 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:38:47.195928 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:38:47.195958 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:38:47.197631 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:38:47.197637 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:38:47.199786 (+ 2149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:38:47.199804 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:38:47.201144 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:38:47.201151 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:38:47.201451 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:47.202300 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:38:47.202332 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:38:47.205293 (+ 2961us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:38:47.209982 (+ 4689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:38:47.209993 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:38:47.210014 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:38:47.210700 (+ 686us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:38:47.211237 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:38:47.211243 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:38:47.211246 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:38:47.211530 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:38:47.212238 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:38:47.212248 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:38:47.212784 (+ 536us) client_negotiation.cc:769] Sending connection context
0620 23:38:47.213202 (+ 418us) client_negotiation.cc:240] Negotiation successful
0620 23:38:47.213874 (+ 672us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12644,"thread_start_us":5443,"threads_started":1}
I20260620 23:38:47.217494 13759 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42337
I20260620 23:38:47.217859 13759 heartbeater.cc:464] Registering TS with master...
I20260620 23:38:47.218971 13759 heartbeater.cc:511] Master 127.0.97.62:42337 requested a full tablet report, sending...
I20260620 23:38:47.221205 13283 ts_manager.cc:194] Registered new tserver with Master: e87feaa0e06d4a858c302e28cd3b0afc (127.0.97.3:38123)
I20260620 23:38:47.222333 13283 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:54715
I20260620 23:38:47.226850 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 23:38:47 dist-test-slave-tcgx krb5kdc[12271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998726, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 23:38:47.257094 13764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:47.237553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58260 (local address 127.0.97.62:42337)
0620 23:38:47.237828 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0620 23:38:47.237836 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:38:47.240018 (+ 2182us) server_negotiation.cc:407] Connection header received
0620 23:38:47.240247 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:38:47.240254 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:38:47.240346 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:38:47.240473 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:38:47.242547 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:47.243678 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:47.245801 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:47.246436 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:47.246546 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:47.250489 (+ 3943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:38:47.250496 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:38:47.250501 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:38:47.250548 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:38:47.254117 (+ 3569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:38:47.254798 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:38:47.254805 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:38:47.254809 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:38:47.254998 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:38:47.255381 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:38:47.255388 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:38:47.255391 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:38:47.255735 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:38:47.255968 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0620 23:38:47.256541 (+ 573us) server_negotiation.cc:299] Negotiation successful
0620 23:38:47.256743 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260620 23:38:47.263685 13283 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58260:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:38:47.275032 13283 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:38:47.319001 13775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:47.302582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33317 (local address 127.0.0.1:46166)
0620 23:38:47.308658 (+ 6076us) negotiation.cc:107] Waiting for socket to connect
0620 23:38:47.308736 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0620 23:38:47.308965 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:38:47.310193 (+ 1228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:38:47.310201 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:38:47.310242 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:38:47.310678 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:38:47.310693 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:38:47.312962 (+ 2269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:38:47.312968 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:38:47.315640 (+ 2672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:38:47.315657 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:38:47.317910 (+ 2253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:38:47.317917 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:38:47.318136 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:47.318197 (+ 61us) client_negotiation.cc:769] Sending connection context
0620 23:38:47.318341 (+ 144us) client_negotiation.cc:240] Negotiation successful
0620 23:38:47.318559 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5781,"thread_start_us":8023,"threads_started":1}
I20260620 23:38:47.319996 13776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:47.303187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46166 (local address 127.0.97.2:33317)
0620 23:38:47.308053 (+ 4866us) server_negotiation.cc:206] Beginning negotiation
0620 23:38:47.308063 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:38:47.309250 (+ 1187us) server_negotiation.cc:407] Connection header received
0620 23:38:47.309364 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:38:47.309373 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:38:47.309665 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:38:47.309892 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:38:47.310911 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:47.312782 (+ 1871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:47.315855 (+ 3073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:47.317347 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:47.317487 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:47.317701 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0620 23:38:47.319484 (+ 1783us) server_negotiation.cc:299] Negotiation successful
0620 23:38:47.319667 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4675,"thread_start_us":4551,"threads_started":1}
I20260620 23:38:47.324288 13560 tablet_service.cc:1467] Processing CreateTablet for tablet 5b4ca17b7aea4cb19b40418c26a3ca0f (DEFAULT_TABLE table=test-table [id=0a6cd08c75ea4f5aa0d0edd22f1d9b65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:38:47.325922 13560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b4ca17b7aea4cb19b40418c26a3ca0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:38:47.331957 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:47.309828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36720 (local address 127.0.97.3:38123)
0620 23:38:47.310438 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0620 23:38:47.310447 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:38:47.321162 (+ 10715us) server_negotiation.cc:407] Connection header received
0620 23:38:47.321728 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:38:47.321741 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:38:47.322050 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:38:47.322242 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:38:47.323375 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:47.325184 (+ 1809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:47.328139 (+ 2955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:47.329958 (+ 1819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:47.330085 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:47.330367 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0620 23:38:47.331282 (+ 915us) server_negotiation.cc:299] Negotiation successful
0620 23:38:47.331512 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":382,"threads_started":1}
I20260620 23:38:47.333309 13777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:47.309219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38123 (local address 127.0.0.1:36720)
0620 23:38:47.321010 (+ 11791us) negotiation.cc:107] Waiting for socket to connect
0620 23:38:47.321033 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 23:38:47.321494 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:38:47.322650 (+ 1156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:38:47.322658 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:38:47.322691 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:38:47.323128 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:38:47.323143 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:38:47.325381 (+ 2238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:38:47.325386 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:38:47.327932 (+ 2546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:38:47.327952 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:38:47.330597 (+ 2645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:38:47.330606 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:38:47.330764 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:47.330797 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 23:38:47.330883 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 23:38:47.331057 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11628,"thread_start_us":11974,"threads_started":1}
I20260620 23:38:47.337783 13694 tablet_service.cc:1467] Processing CreateTablet for tablet 5b4ca17b7aea4cb19b40418c26a3ca0f (DEFAULT_TABLE table=test-table [id=0a6cd08c75ea4f5aa0d0edd22f1d9b65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:38:47.341082 13694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b4ca17b7aea4cb19b40418c26a3ca0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:38:47.345644 13781 tablet_bootstrap.cc:492] T 5b4ca17b7aea4cb19b40418c26a3ca0f P deed2cf1304549448c299f8d37361198: Bootstrap starting.
I20260620 23:38:47.349918 13781 tablet_bootstrap.cc:654] T 5b4ca17b7aea4cb19b40418c26a3ca0f P deed2cf1304549448c299f8d37361198: Neither blocks nor log segments found. Creating new log.
I20260620 23:38:47.351284 13781 log.cc:824] T 5b4ca17b7aea4cb19b40418c26a3ca0f P deed2cf1304549448c299f8d37361198: Log is configured to *not* fsync() on all Append() calls
I20260620 23:38:47.353880 13775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:47.314426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41993 (local address 127.0.0.1:52936)
0620 23:38:47.331259 (+ 16833us) negotiation.cc:107] Waiting for socket to connect
0620 23:38:47.331281 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 23:38:47.331390 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:38:47.344202 (+ 12812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:38:47.344208 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:38:47.344258 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:38:47.344459 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:38:47.344472 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:38:47.349108 (+ 4636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:38:47.349114 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:38:47.351099 (+ 1985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:38:47.351118 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:38:47.353078 (+ 1960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:38:47.353086 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:38:47.353293 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:47.353332 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 23:38:47.353408 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 23:38:47.353567 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16680,"thread_start_us":30404,"threads_started":1}
I20260620 23:38:47.355041 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:47.315215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52936 (local address 127.0.97.1:41993)
0620 23:38:47.343478 (+ 28263us) server_negotiation.cc:206] Beginning negotiation
0620 23:38:47.343487 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:38:47.343519 (+ 32us) server_negotiation.cc:407] Connection header received
0620 23:38:47.343622 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:38:47.343630 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:38:47.343980 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:38:47.344196 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:38:47.344681 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:47.346261 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:47.351343 (+ 5082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:47.352880 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:47.354159 (+ 1279us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:47.354412 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0620 23:38:47.354492 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 23:38:47.354659 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28069,"thread_start_us":29171,"threads_started":1}
I20260620 23:38:47.359480 13416 tablet_service.cc:1467] Processing CreateTablet for tablet 5b4ca17b7aea4cb19b40418c26a3ca0f (DEFAULT_TABLE table=test-table [id=0a6cd08c75ea4f5aa0d0edd22f1d9b65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:38:47.361168 13416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b4ca17b7aea4cb19b40418c26a3ca0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:38:47.365617 13781 tablet_bootstrap.cc:492] T 5b4ca17b7aea4cb19b40418c26a3ca0f P deed2cf1304549448c299f8d37361198: No bootstrap required, opened a new log
I20260620 23:38:47.366014 13781 ts_tablet_manager.cc:1397] T 5b4ca17b7aea4cb19b40418c26a3ca0f P deed2cf1304549448c299f8d37361198: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.008s
I20260620 23:38:47.377640 13783 tablet_bootstrap.cc:492] T 5b4ca17b7aea4cb19b40418c26a3ca0f P e87feaa0e06d4a858c302e28cd3b0afc: Bootstrap starting.
I20260620 23:38:47.382628 13783 tablet_bootstrap.cc:654] T 5b4ca17b7aea4cb19b40418c26a3ca0f P e87feaa0e06d4a858c302e28cd3b0afc: Neither blocks nor log segments found. Creating new log.
I20260620 23:38:47.388103 13784 tablet_bootstrap.cc:492] T 5b4ca17b7aea4cb19b40418c26a3ca0f P b36b856889874cdfaa5ef8af26462d5c: Bootstrap starting.
I20260620 23:38:47.392272 13783 log.cc:824] T 5b4ca17b7aea4cb19b40418c26a3ca0f P e87feaa0e06d4a858c302e28cd3b0afc: Log is configured to *not* fsync() on all Append() calls
I20260620 23:38:47.394227 13784 tablet_bootstrap.cc:654] T 5b4ca17b7aea4cb19b40418c26a3ca0f P b36b856889874cdfaa5ef8af26462d5c: Neither blocks nor log segments found. Creating new log.
I20260620 23:38:47.395681 13784 log.cc:824] T 5b4ca17b7aea4cb19b40418c26a3ca0f P b36b856889874cdfaa5ef8af26462d5c: Log is configured to *not* fsync() on all Append() calls
I20260620 23:38:47.403609 13781 raft_consensus.cc:348] T 5b4ca17b7aea4cb19b40418c26a3ca0f P deed2cf1304549448c299f8d37361198 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deed2cf1304549448c299f8d37361198" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33317 } } peers { permanent_uuid: "e87feaa0e06d4a858c302e28cd3b0afc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38123 } } peers { permanent_uuid: "b36b856889874cdfaa5ef8af26462d5c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41993 } }
I20260620 23:38:47.405037 13781 raft_consensus.cc:374] T 5b4ca17b7aea4cb19b40418c26a3ca0f P deed2cf1304549448c299f8d37361198 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:38:47.405292 13781 raft_consensus.cc:729] T 5b4ca17b7aea4cb19b40418c26a3ca0f P deed2cf1304549448c299f8d37361198 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: deed2cf1304549448c299f8d37361198, State: Initialized, Role: FOLLOWER
I20260620 23:38:47.406445 13781 consensus_queue.cc:260] T 5b4ca17b7aea4cb19b40418c26a3ca0f P deed2cf1304549448c299f8d37361198 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deed2cf1304549448c299f8d37361198" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33317 } } peers { permanent_uuid: "e87feaa0e06d4a858c302e28cd3b0afc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38123 } } peers { permanent_uuid: "b36b856889874cdfaa5ef8af26462d5c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41993 } }
I20260620 23:38:47.410993 13783 tablet_bootstrap.cc:492] T 5b4ca17b7aea4cb19b40418c26a3ca0f P e87feaa0e06d4a858c302e28cd3b0afc: No bootstrap required, opened a new log
I20260620 23:38:47.411403 13783 ts_tablet_manager.cc:1397] T 5b4ca17b7aea4cb19b40418c26a3ca0f P e87feaa0e06d4a858c302e28cd3b0afc: Time spent bootstrapping tablet: real 0.034s user 0.015s sys 0.000s
I20260620 23:38:47.413883 13781 ts_tablet_manager.cc:1428] T 5b4ca17b7aea4cb19b40418c26a3ca0f P deed2cf1304549448c299f8d37361198: Time spent starting tablet: real 0.039s user 0.009s sys 0.005s
I20260620 23:38:47.414335 13625 heartbeater.cc:503] Master 127.0.97.62:42337 was elected leader, sending a full tablet report...
I20260620 23:38:47.424556 13784 tablet_bootstrap.cc:492] T 5b4ca17b7aea4cb19b40418c26a3ca0f P b36b856889874cdfaa5ef8af26462d5c: No bootstrap required, opened a new log
I20260620 23:38:47.424911 13784 ts_tablet_manager.cc:1397] T 5b4ca17b7aea4cb19b40418c26a3ca0f P b36b856889874cdfaa5ef8af26462d5c: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.010s
I20260620 23:38:47.426157 13783 raft_consensus.cc:348] T 5b4ca17b7aea4cb19b40418c26a3ca0f P e87feaa0e06d4a858c302e28cd3b0afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deed2cf1304549448c299f8d37361198" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33317 } } peers { permanent_uuid: "e87feaa0e06d4a858c302e28cd3b0afc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38123 } } peers { permanent_uuid: "b36b856889874cdfaa5ef8af26462d5c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41993 } }
I20260620 23:38:47.426715 13783 raft_consensus.cc:374] T 5b4ca17b7aea4cb19b40418c26a3ca0f P e87feaa0e06d4a858c302e28cd3b0afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:38:47.426842 13783 raft_consensus.cc:729] T 5b4ca17b7aea4cb19b40418c26a3ca0f P e87feaa0e06d4a858c302e28cd3b0afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e87feaa0e06d4a858c302e28cd3b0afc, State: Initialized, Role: FOLLOWER
I20260620 23:38:47.427628 13783 consensus_queue.cc:260] T 5b4ca17b7aea4cb19b40418c26a3ca0f P e87feaa0e06d4a858c302e28cd3b0afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deed2cf1304549448c299f8d37361198" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33317 } } peers { permanent_uuid: "e87feaa0e06d4a858c302e28cd3b0afc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38123 } } peers { permanent_uuid: "b36b856889874cdfaa5ef8af26462d5c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41993 } }
I20260620 23:38:47.436378 13784 raft_consensus.cc:348] T 5b4ca17b7aea4cb19b40418c26a3ca0f P b36b856889874cdfaa5ef8af26462d5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deed2cf1304549448c299f8d37361198" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33317 } } peers { permanent_uuid: "e87feaa0e06d4a858c302e28cd3b0afc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38123 } } peers { permanent_uuid: "b36b856889874cdfaa5ef8af26462d5c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41993 } }
I20260620 23:38:47.436952 13784 raft_consensus.cc:374] T 5b4ca17b7aea4cb19b40418c26a3ca0f P b36b856889874cdfaa5ef8af26462d5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:38:47.437067 13784 raft_consensus.cc:729] T 5b4ca17b7aea4cb19b40418c26a3ca0f P b36b856889874cdfaa5ef8af26462d5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b36b856889874cdfaa5ef8af26462d5c, State: Initialized, Role: FOLLOWER
I20260620 23:38:47.437893 13784 consensus_queue.cc:260] T 5b4ca17b7aea4cb19b40418c26a3ca0f P b36b856889874cdfaa5ef8af26462d5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deed2cf1304549448c299f8d37361198" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33317 } } peers { permanent_uuid: "e87feaa0e06d4a858c302e28cd3b0afc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38123 } } peers { permanent_uuid: "b36b856889874cdfaa5ef8af26462d5c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41993 } }
I20260620 23:38:47.439786 13759 heartbeater.cc:503] Master 127.0.97.62:42337 was elected leader, sending a full tablet report...
I20260620 23:38:47.440577 13783 ts_tablet_manager.cc:1428] T 5b4ca17b7aea4cb19b40418c26a3ca0f P e87feaa0e06d4a858c302e28cd3b0afc: Time spent starting tablet: real 0.029s user 0.010s sys 0.013s
W20260620 23:38:47.441946 13762 tablet.cc:2369] T 5b4ca17b7aea4cb19b40418c26a3ca0f P e87feaa0e06d4a858c302e28cd3b0afc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:38:47.446267 13481 heartbeater.cc:503] Master 127.0.97.62:42337 was elected leader, sending a full tablet report...
I20260620 23:38:47.447279 13784 ts_tablet_manager.cc:1428] T 5b4ca17b7aea4cb19b40418c26a3ca0f P b36b856889874cdfaa5ef8af26462d5c: Time spent starting tablet: real 0.022s user 0.017s sys 0.006s
W20260620 23:38:47.527631 13482 tablet.cc:2369] T 5b4ca17b7aea4cb19b40418c26a3ca0f P b36b856889874cdfaa5ef8af26462d5c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:38:47.586217 13788 raft_consensus.cc:482] T 5b4ca17b7aea4cb19b40418c26a3ca0f P e87feaa0e06d4a858c302e28cd3b0afc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:38:47.586549 13788 raft_consensus.cc:504] T 5b4ca17b7aea4cb19b40418c26a3ca0f P e87feaa0e06d4a858c302e28cd3b0afc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deed2cf1304549448c299f8d37361198" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33317 } } peers { permanent_uuid: "e87feaa0e06d4a858c302e28cd3b0afc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38123 } } peers { permanent_uuid: "b36b856889874cdfaa5ef8af26462d5c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41993 } }
I20260620 23:38:47.588439 13788 leader_election.cc:288] T 5b4ca17b7aea4cb19b40418c26a3ca0f P e87feaa0e06d4a858c302e28cd3b0afc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers deed2cf1304549448c299f8d37361198 (127.0.97.2:33317), b36b856889874cdfaa5ef8af26462d5c (127.0.97.1:41993)
I20260620 23:38:47.595543 13763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:47.588975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41993 (local address 127.0.97.3:52325)
0620 23:38:47.589294 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0620 23:38:47.589315 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:38:47.589469 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:38:47.589929 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:38:47.589935 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:38:47.589964 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:38:47.590184 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:38:47.590196 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:38:47.591647 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:38:47.591653 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:38:47.593391 (+ 1738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:38:47.593403 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:38:47.594578 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:38:47.594585 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:38:47.594869 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:47.594986 (+ 117us) client_negotiation.cc:769] Sending connection context
0620 23:38:47.595101 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 23:38:47.595246 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"mutex_wait_us":113}
I20260620 23:38:47.595532 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:47.588994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52325 (local address 127.0.97.1:41993)
0620 23:38:47.589216 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 23:38:47.589222 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:38:47.589413 (+ 191us) server_negotiation.cc:407] Connection header received
0620 23:38:47.589646 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:38:47.589652 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:38:47.589763 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:38:47.589895 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:38:47.590380 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:47.591516 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:47.593571 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:47.594384 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:47.594546 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:47.594657 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 23:38:47.595143 (+ 486us) server_negotiation.cc:299] Negotiation successful
0620 23:38:47.595287 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 23:38:47.596879 13436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b4ca17b7aea4cb19b40418c26a3ca0f" candidate_uuid: "e87feaa0e06d4a858c302e28cd3b0afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b36b856889874cdfaa5ef8af26462d5c" is_pre_election: true
I20260620 23:38:47.597442 13776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:47.588994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58867 (local address 127.0.97.2:33317)
0620 23:38:47.589207 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 23:38:47.589215 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:38:47.590595 (+ 1380us) server_negotiation.cc:407] Connection header received
0620 23:38:47.590774 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:38:47.590778 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:38:47.590848 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:38:47.590991 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:38:47.591607 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:47.592700 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:47.595197 (+ 2497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:47.596504 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:47.596626 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:47.596713 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 23:38:47.597056 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 23:38:47.597194 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 23:38:47.597517 13797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:47.589584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33317 (local address 127.0.97.3:58867)
0620 23:38:47.590478 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0620 23:38:47.590494 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:38:47.590635 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:38:47.591119 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:38:47.591124 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:38:47.591148 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:38:47.591447 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:38:47.591454 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:38:47.592897 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:38:47.592902 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:38:47.594955 (+ 2053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:38:47.594974 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:38:47.596682 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:38:47.596688 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:38:47.596860 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:47.596898 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 23:38:47.597002 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 23:38:47.597156 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":775,"mutex_wait_us":37,"thread_start_us":583,"threads_started":1}
I20260620 23:38:47.597496 13436 raft_consensus.cc:2454] T 5b4ca17b7aea4cb19b40418c26a3ca0f P b36b856889874cdfaa5ef8af26462d5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e87feaa0e06d4a858c302e28cd3b0afc in term 0.
I20260620 23:38:47.598347 13647 leader_election.cc:302] T 5b4ca17b7aea4cb19b40418c26a3ca0f P e87feaa0e06d4a858c302e28cd3b0afc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b36b856889874cdfaa5ef8af26462d5c, e87feaa0e06d4a858c302e28cd3b0afc; no voters:
I20260620 23:38:47.598765 13788 raft_consensus.cc:2781] T 5b4ca17b7aea4cb19b40418c26a3ca0f P e87feaa0e06d4a858c302e28cd3b0afc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:38:47.598644 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b4ca17b7aea4cb19b40418c26a3ca0f" candidate_uuid: "e87feaa0e06d4a858c302e28cd3b0afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "deed2cf1304549448c299f8d37361198" is_pre_election: true
I20260620 23:38:47.598930 13788 raft_consensus.cc:482] T 5b4ca17b7aea4cb19b40418c26a3ca0f P e87feaa0e06d4a858c302e28cd3b0afc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:38:47.599071 13788 raft_consensus.cc:3037] T 5b4ca17b7aea4cb19b40418c26a3ca0f P e87feaa0e06d4a858c302e28cd3b0afc [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:38:47.599191 13580 raft_consensus.cc:2454] T 5b4ca17b7aea4cb19b40418c26a3ca0f P deed2cf1304549448c299f8d37361198 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e87feaa0e06d4a858c302e28cd3b0afc in term 0.
I20260620 23:38:47.601585 13788 raft_consensus.cc:504] T 5b4ca17b7aea4cb19b40418c26a3ca0f P e87feaa0e06d4a858c302e28cd3b0afc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deed2cf1304549448c299f8d37361198" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33317 } } peers { permanent_uuid: "e87feaa0e06d4a858c302e28cd3b0afc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38123 } } peers { permanent_uuid: "b36b856889874cdfaa5ef8af26462d5c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41993 } }
I20260620 23:38:47.602241 13788 leader_election.cc:288] T 5b4ca17b7aea4cb19b40418c26a3ca0f P e87feaa0e06d4a858c302e28cd3b0afc [CANDIDATE]: Term 1 election: Requested vote from peers deed2cf1304549448c299f8d37361198 (127.0.97.2:33317), b36b856889874cdfaa5ef8af26462d5c (127.0.97.1:41993)
I20260620 23:38:47.602790 13436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b4ca17b7aea4cb19b40418c26a3ca0f" candidate_uuid: "e87feaa0e06d4a858c302e28cd3b0afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b36b856889874cdfaa5ef8af26462d5c"
I20260620 23:38:47.602999 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b4ca17b7aea4cb19b40418c26a3ca0f" candidate_uuid: "e87feaa0e06d4a858c302e28cd3b0afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "deed2cf1304549448c299f8d37361198"
I20260620 23:38:47.603165 13436 raft_consensus.cc:3037] T 5b4ca17b7aea4cb19b40418c26a3ca0f P b36b856889874cdfaa5ef8af26462d5c [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:38:47.603243 13580 raft_consensus.cc:3037] T 5b4ca17b7aea4cb19b40418c26a3ca0f P deed2cf1304549448c299f8d37361198 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:38:47.606249 13580 raft_consensus.cc:2454] T 5b4ca17b7aea4cb19b40418c26a3ca0f P deed2cf1304549448c299f8d37361198 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e87feaa0e06d4a858c302e28cd3b0afc in term 1.
I20260620 23:38:47.606690 13649 leader_election.cc:302] T 5b4ca17b7aea4cb19b40418c26a3ca0f P e87feaa0e06d4a858c302e28cd3b0afc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: deed2cf1304549448c299f8d37361198, e87feaa0e06d4a858c302e28cd3b0afc; no voters:
I20260620 23:38:47.607091 13788 raft_consensus.cc:2781] T 5b4ca17b7aea4cb19b40418c26a3ca0f P e87feaa0e06d4a858c302e28cd3b0afc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:38:47.607407 13436 raft_consensus.cc:2454] T 5b4ca17b7aea4cb19b40418c26a3ca0f P b36b856889874cdfaa5ef8af26462d5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e87feaa0e06d4a858c302e28cd3b0afc in term 1.
I20260620 23:38:47.607749 13788 raft_consensus.cc:686] T 5b4ca17b7aea4cb19b40418c26a3ca0f P e87feaa0e06d4a858c302e28cd3b0afc [term 1 LEADER]: Becoming Leader. State: Replica: e87feaa0e06d4a858c302e28cd3b0afc, State: Running, Role: LEADER
I20260620 23:38:47.608562 13788 consensus_queue.cc:237] T 5b4ca17b7aea4cb19b40418c26a3ca0f P e87feaa0e06d4a858c302e28cd3b0afc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deed2cf1304549448c299f8d37361198" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33317 } } peers { permanent_uuid: "e87feaa0e06d4a858c302e28cd3b0afc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38123 } } peers { permanent_uuid: "b36b856889874cdfaa5ef8af26462d5c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41993 } }
I20260620 23:38:47.613754 13283 catalog_manager.cc:5495] T 5b4ca17b7aea4cb19b40418c26a3ca0f P e87feaa0e06d4a858c302e28cd3b0afc reported cstate change: term changed from 0 to 1, leader changed from <none> to e87feaa0e06d4a858c302e28cd3b0afc (127.0.97.3). New cstate: current_term: 1 leader_uuid: "e87feaa0e06d4a858c302e28cd3b0afc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deed2cf1304549448c299f8d37361198" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e87feaa0e06d4a858c302e28cd3b0afc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b36b856889874cdfaa5ef8af26462d5c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41993 } health_report { overall_health: UNKNOWN } } }
W20260620 23:38:47.663853 13626 tablet.cc:2369] T 5b4ca17b7aea4cb19b40418c26a3ca0f P deed2cf1304549448c299f8d37361198: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:38:47.689890 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:47.683494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36726 (local address 127.0.97.3:38123)
0620 23:38:47.683725 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 23:38:47.683732 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:38:47.683753 (+ 21us) server_negotiation.cc:407] Connection header received
0620 23:38:47.683892 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:38:47.683898 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:38:47.684015 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:38:47.684181 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:38:47.684536 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:47.685636 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:47.687249 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:47.687821 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:47.687944 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:47.688489 (+ 545us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:38:47.689229 (+ 740us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:38:47.689378 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 23:38:47.689515 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 23:38:47.689604 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 23:38:47.699247 13580 raft_consensus.cc:1261] T 5b4ca17b7aea4cb19b40418c26a3ca0f P deed2cf1304549448c299f8d37361198 [term 1 FOLLOWER]: Refusing update from remote peer e87feaa0e06d4a858c302e28cd3b0afc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:38:47.699688 13436 raft_consensus.cc:1261] T 5b4ca17b7aea4cb19b40418c26a3ca0f P b36b856889874cdfaa5ef8af26462d5c [term 1 FOLLOWER]: Refusing update from remote peer e87feaa0e06d4a858c302e28cd3b0afc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:38:47.700477 13788 consensus_queue.cc:1035] T 5b4ca17b7aea4cb19b40418c26a3ca0f P e87feaa0e06d4a858c302e28cd3b0afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "deed2cf1304549448c299f8d37361198" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:38:47.700996 13798 consensus_queue.cc:1035] T 5b4ca17b7aea4cb19b40418c26a3ca0f P e87feaa0e06d4a858c302e28cd3b0afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b36b856889874cdfaa5ef8af26462d5c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:38:47.720363 13801 mvcc.cc:205] Tried to move back new op lower bound from 7299066788643913728 to 7299066788288757760. Current Snapshot: MvccSnapshot[applied={T|T < 7299066788643913728}]
I20260620 23:38:47.723686 13803 mvcc.cc:205] Tried to move back new op lower bound from 7299066788643913728 to 7299066788288757760. Current Snapshot: MvccSnapshot[applied={T|T < 7299066788643913728}]
I20260620 23:38:47.746390 13282 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58260:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:38:47.746820 13282 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58260:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:38:47.750835 13282 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3b3f964a0df440dcbc394fedf4b52796: Sending DeleteTablet for 3 replicas of tablet 5b4ca17b7aea4cb19b40418c26a3ca0f
I20260620 23:38:47.752213 13560 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b4ca17b7aea4cb19b40418c26a3ca0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:38:47 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46166
I20260620 23:38:47.752534 13416 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b4ca17b7aea4cb19b40418c26a3ca0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:38:47 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52936
I20260620 23:38:47.753023 13694 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b4ca17b7aea4cb19b40418c26a3ca0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:38:47 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36720
I20260620 23:38:47.753696 13811 tablet_replica.cc:330] T 5b4ca17b7aea4cb19b40418c26a3ca0f P deed2cf1304549448c299f8d37361198: stopping tablet replica
I20260620 23:38:47.754145 13811 raft_consensus.cc:2229] T 5b4ca17b7aea4cb19b40418c26a3ca0f P deed2cf1304549448c299f8d37361198 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:38:47.754678 13811 raft_consensus.cc:2258] T 5b4ca17b7aea4cb19b40418c26a3ca0f P deed2cf1304549448c299f8d37361198 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:38:47.756078 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 13320
I20260620 23:38:47.764681 13813 tablet_replica.cc:330] T 5b4ca17b7aea4cb19b40418c26a3ca0f P e87feaa0e06d4a858c302e28cd3b0afc: stopping tablet replica
I20260620 23:38:47.765260 13813 raft_consensus.cc:2229] T 5b4ca17b7aea4cb19b40418c26a3ca0f P e87feaa0e06d4a858c302e28cd3b0afc [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:38:47.766038 13813 raft_consensus.cc:2258] T 5b4ca17b7aea4cb19b40418c26a3ca0f P e87feaa0e06d4a858c302e28cd3b0afc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:38:47.768937 13813 ts_tablet_manager.cc:1905] T 5b4ca17b7aea4cb19b40418c26a3ca0f P e87feaa0e06d4a858c302e28cd3b0afc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:38:47.768959 13811 ts_tablet_manager.cc:1905] T 5b4ca17b7aea4cb19b40418c26a3ca0f P deed2cf1304549448c299f8d37361198: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:38:47.776240 13811 ts_tablet_manager.cc:1918] T 5b4ca17b7aea4cb19b40418c26a3ca0f P deed2cf1304549448c299f8d37361198: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:38:47.776444 13811 log.cc:1196] T 5b4ca17b7aea4cb19b40418c26a3ca0f P deed2cf1304549448c299f8d37361198: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-1/wal/wals/5b4ca17b7aea4cb19b40418c26a3ca0f
I20260620 23:38:47.776584 13813 ts_tablet_manager.cc:1918] T 5b4ca17b7aea4cb19b40418c26a3ca0f P e87feaa0e06d4a858c302e28cd3b0afc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:38:47.776774 13813 log.cc:1196] T 5b4ca17b7aea4cb19b40418c26a3ca0f P e87feaa0e06d4a858c302e28cd3b0afc: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-2/wal/wals/5b4ca17b7aea4cb19b40418c26a3ca0f
I20260620 23:38:47.776954 13812 tablet_replica.cc:330] T 5b4ca17b7aea4cb19b40418c26a3ca0f P b36b856889874cdfaa5ef8af26462d5c: stopping tablet replica
I20260620 23:38:47.777076 13811 ts_tablet_manager.cc:1939] T 5b4ca17b7aea4cb19b40418c26a3ca0f P deed2cf1304549448c299f8d37361198: Deleting consensus metadata
I20260620 23:38:47.777338 13813 ts_tablet_manager.cc:1939] T 5b4ca17b7aea4cb19b40418c26a3ca0f P e87feaa0e06d4a858c302e28cd3b0afc: Deleting consensus metadata
I20260620 23:38:47.777724 13812 raft_consensus.cc:2229] T 5b4ca17b7aea4cb19b40418c26a3ca0f P b36b856889874cdfaa5ef8af26462d5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:38:47.778425 13812 raft_consensus.cc:2258] T 5b4ca17b7aea4cb19b40418c26a3ca0f P b36b856889874cdfaa5ef8af26462d5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:38:47.779353 13269 catalog_manager.cc:4841] TS deed2cf1304549448c299f8d37361198 (127.0.97.2:33317): tablet 5b4ca17b7aea4cb19b40418c26a3ca0f (table test-table [id=0a6cd08c75ea4f5aa0d0edd22f1d9b65]) successfully deleted
I20260620 23:38:47.779304 13270 catalog_manager.cc:4841] TS e87feaa0e06d4a858c302e28cd3b0afc (127.0.97.3:38123): tablet 5b4ca17b7aea4cb19b40418c26a3ca0f (table test-table [id=0a6cd08c75ea4f5aa0d0edd22f1d9b65]) successfully deleted
I20260620 23:38:47.781224 13812 ts_tablet_manager.cc:1905] T 5b4ca17b7aea4cb19b40418c26a3ca0f P b36b856889874cdfaa5ef8af26462d5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:38:47.788607 13812 ts_tablet_manager.cc:1918] T 5b4ca17b7aea4cb19b40418c26a3ca0f P b36b856889874cdfaa5ef8af26462d5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:38:47.788830 13812 log.cc:1196] T 5b4ca17b7aea4cb19b40418c26a3ca0f P b36b856889874cdfaa5ef8af26462d5c: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998525501163-388-0/minicluster-data/ts-0/wal/wals/5b4ca17b7aea4cb19b40418c26a3ca0f
I20260620 23:38:47.789434 13812 ts_tablet_manager.cc:1939] T 5b4ca17b7aea4cb19b40418c26a3ca0f P b36b856889874cdfaa5ef8af26462d5c: Deleting consensus metadata
I20260620 23:38:47.791720 13267 catalog_manager.cc:4841] TS b36b856889874cdfaa5ef8af26462d5c (127.0.97.1:41993): tablet 5b4ca17b7aea4cb19b40418c26a3ca0f (table test-table [id=0a6cd08c75ea4f5aa0d0edd22f1d9b65]) successfully deleted
I20260620 23:38:47.795578 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:47.760109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52940 (local address 127.0.97.1:41993)
0620 23:38:47.760362 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0620 23:38:47.760371 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:38:47.764926 (+ 4555us) server_negotiation.cc:407] Connection header received
0620 23:38:47.765148 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:38:47.765156 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:38:47.765287 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:38:47.765419 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:38:47.780947 (+ 15528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:47.782059 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:47.784047 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:47.784741 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:47.784886 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:47.788210 (+ 3324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:38:47.788231 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:38:47.788263 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:38:47.788314 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:38:47.792443 (+ 4129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:38:47.793290 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:38:47.793297 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:38:47.793300 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:38:47.793432 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:38:47.793874 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:38:47.793879 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:38:47.793887 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:38:47.794249 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:38:47.794579 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0620 23:38:47.794928 (+ 349us) server_negotiation.cc:299] Negotiation successful
0620 23:38:47.795187 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 23:38:47.796325 13476 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:38:47.890170 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 13320
I20260620 23:38:47.913419 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 13485
I20260620 23:38:47.929452 13776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:47.914308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46170 (local address 127.0.97.2:33317)
0620 23:38:47.914494 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 23:38:47.914499 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:38:47.914517 (+ 18us) server_negotiation.cc:407] Connection header received
0620 23:38:47.914775 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:38:47.914779 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:38:47.914886 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:38:47.915035 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:38:47.916755 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:47.917455 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:47.919333 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:47.919767 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:47.919872 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:47.923202 (+ 3330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:38:47.923219 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:38:47.923235 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:38:47.923271 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:38:47.926313 (+ 3042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:38:47.927487 (+ 1174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:38:47.927495 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:38:47.927498 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:38:47.927634 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:38:47.928042 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:38:47.928045 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:38:47.928047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:38:47.928391 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:38:47.928604 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 23:38:47.928974 (+ 370us) server_negotiation.cc:299] Negotiation successful
0620 23:38:47.929140 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 23:38:47.930143 13620 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:38:48.052055 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 13485
I20260620 23:38:48.076254 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 13629
I20260620 23:38:48.092412 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:48.077188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36742 (local address 127.0.97.3:38123)
0620 23:38:48.077439 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0620 23:38:48.077446 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:38:48.077472 (+ 26us) server_negotiation.cc:407] Connection header received
0620 23:38:48.077666 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:38:48.077670 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:38:48.077745 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:38:48.077881 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:38:48.079696 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:48.080528 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:48.082466 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:48.082920 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:48.083100 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:48.085940 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:38:48.085997 (+ 57us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:38:48.086016 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:38:48.086068 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:38:48.089368 (+ 3300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:38:48.090078 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:38:48.090085 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:38:48.090088 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:38:48.090250 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:38:48.090664 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:38:48.090670 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:38:48.090673 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:38:48.091171 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:38:48.091488 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0620 23:38:48.091787 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 23:38:48.092012 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 23:38:48.093274 13754 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:38:48.218757 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 13629
I20260620 23:38:48.237615 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 13250
I20260620 23:38:48.238909 13313 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:38:48.353896 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 13250
I20260620 23:38:48.379415 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 23:38:48.715513 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 23:38:48.715739 388 mini_ranger.cc:66] Stopping Ranger...
I20260620 23:38:49.062150 388 mini_ranger.cc:68] Stopped Ranger
2026-06-20 23:38:49.062 UTC [12301] LOG: received smart shutdown request
2026-06-20 23:38:49.065 UTC [12301] LOG: background worker "logical replication launcher" (PID 12310) exited with exit code 1
2026-06-20 23:38:49.065 UTC [12305] LOG: shutting down
2026-06-20 23:38:49.242 UTC [12301] LOG: database system is shut down
2026-06-20T23:38:49Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (53613 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:38:49 dist-test-slave-tcgx krb5kdc[13827](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:38:49 dist-test-slave-tcgx krb5kdc[13827](info): set up 2 sockets
Jun 20 23:38:49 dist-test-slave-tcgx krb5kdc[13827](info): commencing operation
krb5kdc: starting...
W20260620 23:38:50.479378 388 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/krb5kdc/test-user.keytab.
Jun 20 23:38:50 dist-test-slave-tcgx krb5kdc[13827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:38:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:38:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 23:38:50.722518 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45075
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45075,127.0.97.61:44191,127.0.97.60:40145
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:38:51.061012 13840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:38:51.061329 13840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:38:51.061435 13840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:38:51.068853 13840 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:38:51.068980 13840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:38:51.069027 13840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:38:51.069098 13840 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:38:51.069144 13840 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:38:51.079262 13840 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45075,127.0.97.61:44191,127.0.97.60:40145
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:38:51.080603 13840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:38:51.081916 13840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:38:51.094177 13845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:38:51.094210 13846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:38:51.094383 13848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:38:51.095165 13840 server_base.cc:1034] running on GCE node
I20260620 23:38:51.095950 13840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:38:51.097643 13840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:38:51.098866 13840 hybrid_clock.cc:648] HybridClock initialized: now 1781998731098808 us; error 58 us; skew 500 ppm
I20260620 23:38:51.099294 13840 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:38:51 dist-test-slave-tcgx krb5kdc[13827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998731, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:38:51.108592 13840 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 23:38:51.110625 13840 webserver.cc:460] Webserver started at http://127.0.97.62:43165/ using document root <none> and password file <none>
I20260620 23:38:51.111505 13840 fs_manager.cc:359] Metadata directory not provided
I20260620 23:38:51.111624 13840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:38:51.111904 13840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:38:51.115159 13840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/minicluster-data/master-0/data/instance:
uuid: "7e0644fc188f48009d8b4189ba90cf15"
format_stamp: "Formatted at 2026-06-20 23:38:51 on dist-test-slave-tcgx"
server_key: "da6d43686036dc336db30f5bb3dc8537"
server_key_iv: "4d5a4b46ff2c9256bfd02d0456dc868d"
server_key_version: "encryptionkey@0"
I20260620 23:38:51.115945 13840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/minicluster-data/master-0/wal/instance:
uuid: "7e0644fc188f48009d8b4189ba90cf15"
format_stamp: "Formatted at 2026-06-20 23:38:51 on dist-test-slave-tcgx"
server_key: "da6d43686036dc336db30f5bb3dc8537"
server_key_iv: "4d5a4b46ff2c9256bfd02d0456dc868d"
server_key_version: "encryptionkey@0"
I20260620 23:38:51.121569 13840 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:38:51.125588 13855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:38:51.127298 13840 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:38:51.127463 13840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/minicluster-data/master-0/wal
uuid: "7e0644fc188f48009d8b4189ba90cf15"
format_stamp: "Formatted at 2026-06-20 23:38:51 on dist-test-slave-tcgx"
server_key: "da6d43686036dc336db30f5bb3dc8537"
server_key_iv: "4d5a4b46ff2c9256bfd02d0456dc868d"
server_key_version: "encryptionkey@0"
I20260620 23:38:51.127650 13840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:38:51.143160 13840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:38:51.148171 13840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:38:51.148443 13840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:38:51.168735 13840 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45075
I20260620 23:38:51.168762 13906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45075 every 8 connection(s)
I20260620 23:38:51.170219 13840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:38:51.174088 13907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:38:51.176653 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 13840
I20260620 23:38:51.176875 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/minicluster-data/master-0/wal/instance
I20260620 23:38:51.177685 388 external_mini_cluster.cc:1428] Setting key f04769424a1cf6194799257199f6af1d
I20260620 23:38:51.184376 13907 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45075 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
Jun 20 23:38:51 dist-test-slave-tcgx krb5kdc[13827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998731, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:38:51.211910 13910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:51.188721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45075 (local address 127.0.0.1:38692)
0620 23:38:51.189686 (+ 965us) negotiation.cc:107] Waiting for socket to connect
0620 23:38:51.189744 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 23:38:51.191940 (+ 2196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:38:51.192901 (+ 961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:38:51.192907 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:38:51.193684 (+ 777us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:38:51.194045 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:38:51.194064 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:38:51.195719 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:38:51.195726 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:38:51.196951 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:38:51.196963 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:38:51.197935 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:38:51.197942 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:38:51.198102 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:51.198804 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:38:51.198829 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:38:51.203646 (+ 4817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:38:51.207318 (+ 3672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:38:51.207332 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:38:51.207346 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:38:51.207777 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:38:51.208238 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:38:51.208243 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:38:51.208246 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:38:51.208397 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:38:51.209240 (+ 843us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:38:51.209249 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:38:51.209467 (+ 218us) client_negotiation.cc:769] Sending connection context
0620 23:38:51.209752 (+ 285us) client_negotiation.cc:240] Negotiation successful
0620 23:38:51.210441 (+ 689us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"mutex_wait_us":158,"thread_start_us":423,"threads_started":1}
I20260620 23:38:51.211941 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:51.188722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38692 (local address 127.0.97.62:45075)
0620 23:38:51.190602 (+ 1880us) server_negotiation.cc:206] Beginning negotiation
0620 23:38:51.190612 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:38:51.190663 (+ 51us) server_negotiation.cc:407] Connection header received
0620 23:38:51.192377 (+ 1714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:38:51.192384 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:38:51.192648 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:38:51.192891 (+ 243us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:38:51.194258 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:51.195558 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:51.197102 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:51.197679 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:51.197898 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:51.203937 (+ 6039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:38:51.203946 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:38:51.203961 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:38:51.203995 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:38:51.207163 (+ 3168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:38:51.207935 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:38:51.207949 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:38:51.207951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:38:51.208111 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:38:51.208521 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:38:51.208526 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:38:51.208529 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:38:51.209077 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:38:51.209311 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0620 23:38:51.209869 (+ 558us) server_negotiation.cc:299] Negotiation successful
0620 23:38:51.210326 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":75,"server-negotiator.queue_time_us":1645,"thread_start_us":1411,"threads_started":1}
I20260620 23:38:51.215476 13907 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44191 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:38:51.216862 13910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:51.216317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44191 (local address 127.0.0.1:50684)
0620 23:38:51.216528 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 23:38:51.216669 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260620 23:38:51.217207 13857 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44191: connect: Connection refused (error 111)
I20260620 23:38:51.219110 13910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:51.218620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44191 (local address 127.0.0.1:50690)
0620 23:38:51.218808 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 23:38:51.218897 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 23:38:51.219316 13907 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44191: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44191: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260620 23:38:51.230906 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:44191
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45075,127.0.97.61:44191,127.0.97.60:40145
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:38:51.268715 13907 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44191 } attempt: 1
I20260620 23:38:51.270008 13910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:51.269490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44191 (local address 127.0.0.1:50698)
0620 23:38:51.269710 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 23:38:51.269849 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260620 23:38:51.271189 13910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:51.270738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44191 (local address 127.0.0.1:50702)
0620 23:38:51.270904 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 23:38:51.271030 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260620 23:38:51.271418 13907 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44191: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44191: connect: Connection refused (error 111)
I20260620 23:38:51.339699 13907 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44191 } attempt: 2
I20260620 23:38:51.341169 13910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:51.340463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44191 (local address 127.0.0.1:50714)
0620 23:38:51.340784 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0620 23:38:51.340942 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260620 23:38:51.342494 13910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:51.341998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44191 (local address 127.0.0.1:50728)
0620 23:38:51.342196 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 23:38:51.342316 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260620 23:38:51.342741 13907 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44191: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44191: connect: Connection refused (error 111)
I20260620 23:38:51.434029 13907 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44191 } attempt: 3
I20260620 23:38:51.435274 13910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:51.434747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44191 (local address 127.0.0.1:50736)
0620 23:38:51.434995 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 23:38:51.435106 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260620 23:38:51.436386 13910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:51.435982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44191 (local address 127.0.0.1:50746)
0620 23:38:51.436132 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 23:38:51.436225 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 23:38:51.436581 13907 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44191: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44191: connect: Connection refused (error 111)
W20260620 23:38:51.575628 13914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:38:51.575968 13914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:38:51.576061 13914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 23:38:51.579869 13907 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44191 } attempt: 4
I20260620 23:38:51.581233 13910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:51.580596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44191 (local address 127.0.0.1:50760)
0620 23:38:51.580950 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0620 23:38:51.581064 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183}
I20260620 23:38:51.582455 13910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:51.581989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44191 (local address 127.0.0.1:50770)
0620 23:38:51.582181 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 23:38:51.582298 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 23:38:51.582831 13907 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44191: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44191: connect: Connection refused (error 111)
W20260620 23:38:51.585422 13914 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:38:51.585565 13914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:38:51.585670 13914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:38:51.585762 13914 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:38:51.585845 13914 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:38:51.598270 13914 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45075,127.0.97.61:44191,127.0.97.60:40145
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:44191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:38:51.599920 13914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:38:51.601411 13914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:38:51.613878 13920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:38:51.613896 13919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:38:51.613976 13922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:38:51.614636 13914 server_base.cc:1034] running on GCE node
I20260620 23:38:51.615556 13914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:38:51.617098 13914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:38:51.618366 13914 hybrid_clock.cc:648] HybridClock initialized: now 1781998731618330 us; error 40 us; skew 500 ppm
I20260620 23:38:51.618808 13914 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:38:51 dist-test-slave-tcgx krb5kdc[13827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998731, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:38:51.622356 13914 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260620 23:38:51.624017 13914 webserver.cc:460] Webserver started at http://127.0.97.61:45401/ using document root <none> and password file <none>
I20260620 23:38:51.624812 13914 fs_manager.cc:359] Metadata directory not provided
I20260620 23:38:51.624964 13914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:38:51.625294 13914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:38:51.628515 13914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/minicluster-data/master-1/data/instance:
uuid: "a5b1f241ade64178938b982d7a9b524d"
format_stamp: "Formatted at 2026-06-20 23:38:51 on dist-test-slave-tcgx"
server_key: "743002cad3c5ab7b0e94831842428d35"
server_key_iv: "ab632ba7edc03518985a8b83593af327"
server_key_version: "encryptionkey@0"
I20260620 23:38:51.629384 13914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/minicluster-data/master-1/wal/instance:
uuid: "a5b1f241ade64178938b982d7a9b524d"
format_stamp: "Formatted at 2026-06-20 23:38:51 on dist-test-slave-tcgx"
server_key: "743002cad3c5ab7b0e94831842428d35"
server_key_iv: "ab632ba7edc03518985a8b83593af327"
server_key_version: "encryptionkey@0"
I20260620 23:38:51.635288 13914 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260620 23:38:51.639153 13929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:38:51.640748 13914 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:38:51.640985 13914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/minicluster-data/master-1/wal
uuid: "a5b1f241ade64178938b982d7a9b524d"
format_stamp: "Formatted at 2026-06-20 23:38:51 on dist-test-slave-tcgx"
server_key: "743002cad3c5ab7b0e94831842428d35"
server_key_iv: "ab632ba7edc03518985a8b83593af327"
server_key_version: "encryptionkey@0"
I20260620 23:38:51.641230 13914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:38:51.665935 13914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:38:51.671119 13914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:38:51.671459 13914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:38:51.692492 13914 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:44191
I20260620 23:38:51.692529 13980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:44191 every 8 connection(s)
I20260620 23:38:51.694051 13914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/minicluster-data/master-1/data/info.pb
I20260620 23:38:51.694566 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 13914
I20260620 23:38:51.694761 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/minicluster-data/master-1/wal/instance
I20260620 23:38:51.695429 388 external_mini_cluster.cc:1428] Setting key 5e1a28e0f9ef815124bea9326868a71f
I20260620 23:38:51.698340 13981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:38:51.706076 13981 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45075 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
Jun 20 23:38:51 dist-test-slave-tcgx krb5kdc[13827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998731, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:38:51.729921 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:51.709138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38698 (local address 127.0.97.62:45075)
0620 23:38:51.709380 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 23:38:51.709387 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:38:51.711042 (+ 1655us) server_negotiation.cc:407] Connection header received
0620 23:38:51.713226 (+ 2184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:38:51.713234 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:38:51.713341 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:38:51.713481 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:38:51.715393 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:51.716148 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:51.718137 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:51.718566 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:51.718724 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:51.723244 (+ 4520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:38:51.723254 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:38:51.723261 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:38:51.723316 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:38:51.726321 (+ 3005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:38:51.727395 (+ 1074us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:38:51.727400 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:38:51.727402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:38:51.727552 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:38:51.728041 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:38:51.728045 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:38:51.728048 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:38:51.728353 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:38:51.728529 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 23:38:51.729318 (+ 789us) server_negotiation.cc:299] Negotiation successful
0620 23:38:51.729576 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260620 23:38:51.731069 13984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:51.709770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45075 (local address 127.0.0.1:38698)
0620 23:38:51.710765 (+ 995us) negotiation.cc:107] Waiting for socket to connect
0620 23:38:51.710842 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0620 23:38:51.712866 (+ 2024us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:38:51.713808 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:38:51.713819 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:38:51.714396 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:38:51.715172 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:38:51.715195 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:38:51.716329 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:38:51.716333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:38:51.717945 (+ 1612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:38:51.717959 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:38:51.718831 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:38:51.718838 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:38:51.719169 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:51.720130 (+ 961us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:38:51.720166 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:38:51.722840 (+ 2674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:38:51.726523 (+ 3683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:38:51.726557 (+ 34us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:38:51.726582 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:38:51.727228 (+ 646us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:38:51.727698 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:38:51.727702 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:38:51.727705 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:38:51.727898 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:38:51.728494 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:38:51.728503 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:38:51.728910 (+ 407us) client_negotiation.cc:769] Sending connection context
0620 23:38:51.729201 (+ 291us) client_negotiation.cc:240] Negotiation successful
0620 23:38:51.729765 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":475,"threads_started":1}
I20260620 23:38:51.733033 13981 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44191 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:38:51 dist-test-slave-tcgx krb5kdc[13827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998731, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260620 23:38:51.749269 13984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:51.733843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44191 (local address 127.0.0.1:50784)
0620 23:38:51.734113 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 23:38:51.734136 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 23:38:51.734260 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:38:51.735392 (+ 1132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:38:51.735396 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:38:51.735858 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:38:51.736000 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:38:51.736010 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:38:51.737394 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:38:51.737402 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:38:51.738264 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:38:51.738275 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:38:51.739214 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:38:51.739219 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:38:51.739480 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:51.740139 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:38:51.740158 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:38:51.742508 (+ 2350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:38:51.746411 (+ 3903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:38:51.746420 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:38:51.746425 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:38:51.747002 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:38:51.747462 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:38:51.747467 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:38:51.747471 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:38:51.747606 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:38:51.748365 (+ 759us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:38:51.748369 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:38:51.748503 (+ 134us) client_negotiation.cc:769] Sending connection context
0620 23:38:51.748617 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 23:38:51.748926 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"mutex_wait_us":26}
I20260620 23:38:51.749693 13986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:51.734149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50784 (local address 127.0.97.61:44191)
0620 23:38:51.734874 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0620 23:38:51.734881 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:38:51.734907 (+ 26us) server_negotiation.cc:407] Connection header received
0620 23:38:51.735010 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:38:51.735018 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:38:51.735233 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:38:51.735411 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:38:51.736165 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:51.737235 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:51.738427 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:51.739067 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:51.739203 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:51.742748 (+ 3545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:38:51.742766 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:38:51.742786 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:38:51.742831 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:38:51.746263 (+ 3432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:38:51.747149 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:38:51.747159 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:38:51.747163 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:38:51.747324 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:38:51.747798 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:38:51.747803 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:38:51.747805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:38:51.748232 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:38:51.748416 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 23:38:51.748811 (+ 395us) server_negotiation.cc:299] Negotiation successful
0620 23:38:51.749146 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":560,"thread_start_us":408,"threads_started":1}
I20260620 23:38:51.750423 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:40145
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45075,127.0.97.61:44191,127.0.97.60:40145
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:38:51.752503 13981 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40145 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:38:51.753789 13984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:51.753287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40145 (local address 127.0.0.1:35918)
0620 23:38:51.753490 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 23:38:51.753620 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260620 23:38:51.754065 13932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40145: connect: Connection refused (error 111)
I20260620 23:38:51.755807 13984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:51.755423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40145 (local address 127.0.0.1:35922)
0620 23:38:51.755609 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 23:38:51.755686 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 23:38:51.756049 13981 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40145: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40145: connect: Connection refused (error 111)
I20260620 23:38:51.805426 13981 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40145 } attempt: 1
I20260620 23:38:51.806981 13984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:51.806309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40145 (local address 127.0.0.1:35924)
0620 23:38:51.806604 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0620 23:38:51.806749 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260620 23:38:51.808314 13984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:51.807891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40145 (local address 127.0.0.1:35926)
0620 23:38:51.808091 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 23:38:51.808165 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260620 23:38:51.808547 13981 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40145: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40145: connect: Connection refused (error 111)
I20260620 23:38:51.876824 13981 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40145 } attempt: 2
I20260620 23:38:51.878044 13984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:51.877514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40145 (local address 127.0.0.1:35930)
0620 23:38:51.877744 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 23:38:51.877839 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260620 23:38:51.879138 13984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:51.878708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40145 (local address 127.0.0.1:35946)
0620 23:38:51.878871 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 23:38:51.879010 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 23:38:51.879359 13981 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40145: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40145: connect: Connection refused (error 111)
I20260620 23:38:51.882095 13907 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44191 } attempt: 5
Jun 20 23:38:51 dist-test-slave-tcgx krb5kdc[13827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998731, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 23:38:51.896835 13986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:51.883029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50786 (local address 127.0.97.61:44191)
0620 23:38:51.883216 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 23:38:51.883221 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:38:51.883260 (+ 39us) server_negotiation.cc:407] Connection header received
0620 23:38:51.883331 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:38:51.883335 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:38:51.883400 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:38:51.883486 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:38:51.884516 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:51.885222 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:51.887056 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:51.887513 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:51.887611 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:51.891393 (+ 3782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:38:51.891399 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:38:51.891402 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:38:51.891437 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:38:51.894113 (+ 2676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:38:51.894831 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:38:51.894836 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:38:51.894838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:38:51.895017 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:38:51.895411 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:38:51.895416 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:38:51.895420 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:38:51.895713 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:38:51.895873 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 23:38:51.896277 (+ 404us) server_negotiation.cc:299] Negotiation successful
0620 23:38:51.896506 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 23:38:51.896837 13910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:51.882790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44191 (local address 127.0.0.1:50786)
0620 23:38:51.883023 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 23:38:51.883044 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:38:51.883175 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:38:51.883558 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:38:51.883563 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:38:51.884102 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:38:51.884327 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:38:51.884341 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:38:51.885399 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:38:51.885408 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:38:51.886816 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:38:51.886837 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:38:51.887672 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:38:51.887678 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:38:51.887913 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:51.888600 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:38:51.888624 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:38:51.891167 (+ 2543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:38:51.894282 (+ 3115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:38:51.894289 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:38:51.894293 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:38:51.894684 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:38:51.895134 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:38:51.895139 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:38:51.895143 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:38:51.895286 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:38:51.895837 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:38:51.895850 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:38:51.896017 (+ 167us) client_negotiation.cc:769] Sending connection context
0620 23:38:51.896160 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 23:38:51.896462 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 23:38:51.898018 13907 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40145 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:38:51.899158 13910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:51.898688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40145 (local address 127.0.0.1:35948)
0620 23:38:51.898870 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 23:38:51.899011 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 23:38:51.900081 13910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:51.899753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40145 (local address 127.0.0.1:35962)
0620 23:38:51.899901 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 23:38:51.899988 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 23:38:51.900272 13907 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40145: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40145: connect: Connection refused (error 111)
I20260620 23:38:51.951505 13907 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40145 } attempt: 1
I20260620 23:38:51.952751 13910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:51.952251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40145 (local address 127.0.0.1:35976)
0620 23:38:51.952483 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0620 23:38:51.952577 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260620 23:38:51.953733 13910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:51.953400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40145 (local address 127.0.0.1:35980)
0620 23:38:51.953547 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 23:38:51.953620 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 23:38:51.953946 13907 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40145: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40145: connect: Connection refused (error 111)
I20260620 23:38:51.970583 13981 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40145 } attempt: 3
I20260620 23:38:51.971786 13984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:51.971324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40145 (local address 127.0.0.1:35992)
0620 23:38:51.971525 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 23:38:51.971624 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260620 23:38:51.972834 13984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:51.972418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40145 (local address 127.0.0.1:35998)
0620 23:38:51.972642 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 23:38:51.972721 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
W20260620 23:38:51.973040 13981 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40145: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40145: connect: Connection refused (error 111)
I20260620 23:38:52.022178 13907 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40145 } attempt: 2
I20260620 23:38:52.023339 13910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:52.022833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40145 (local address 127.0.0.1:36010)
0620 23:38:52.023087 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0620 23:38:52.023170 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260620 23:38:52.024292 13910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:52.023980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40145 (local address 127.0.0.1:36012)
0620 23:38:52.024131 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 23:38:52.024199 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 23:38:52.024489 13907 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40145: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40145: connect: Connection refused (error 111)
W20260620 23:38:52.088981 13988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:38:52.089380 13988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:38:52.089542 13988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:38:52.097174 13988 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:38:52.097358 13988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:38:52.097468 13988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:38:52.097568 13988 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:38:52.097649 13988 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:38:52.108319 13988 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45075,127.0.97.61:44191,127.0.97.60:40145
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:40145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:38:52.109815 13988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:38:52.111300 13988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 23:38:52.116271 13981 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40145 } attempt: 4
I20260620 23:38:52.117792 13984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:52.117038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40145 (local address 127.0.0.1:36022)
0620 23:38:52.117426 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0620 23:38:52.117554 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195}
I20260620 23:38:52.119066 13984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:52.118641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40145 (local address 127.0.0.1:36030)
0620 23:38:52.118807 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 23:38:52.118888 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 23:38:52.119279 13981 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40145: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40145: connect: Connection refused (error 111)
W20260620 23:38:52.124369 13993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:38:52.124280 13994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:38:52.124246 13996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:38:52.125419 13988 server_base.cc:1034] running on GCE node
I20260620 23:38:52.126269 13988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:38:52.127960 13988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:38:52.129220 13988 hybrid_clock.cc:648] HybridClock initialized: now 1781998732129172 us; error 44 us; skew 500 ppm
I20260620 23:38:52.129616 13988 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:38:52.130697 13907 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40145 } attempt: 3
I20260620 23:38:52.131925 13910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:52.131341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40145 (local address 127.0.0.1:36042)
0620 23:38:52.131608 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 23:38:52.131715 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
Jun 20 23:38:52 dist-test-slave-tcgx krb5kdc[13827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998732, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:38:52.133131 13910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:52.132676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40145 (local address 127.0.0.1:36048)
0620 23:38:52.132928 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 23:38:52.133013 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
W20260620 23:38:52.133419 13907 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40145: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40145: connect: Connection refused (error 111)
I20260620 23:38:52.133803 13988 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260620 23:38:52.135419 13988 webserver.cc:460] Webserver started at http://127.0.97.60:39667/ using document root <none> and password file <none>
I20260620 23:38:52.136137 13988 fs_manager.cc:359] Metadata directory not provided
I20260620 23:38:52.136240 13988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:38:52.136500 13988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:38:52.139710 13988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/minicluster-data/master-2/data/instance:
uuid: "b25ed4cd67af4c2383c7bdda1640d0e4"
format_stamp: "Formatted at 2026-06-20 23:38:52 on dist-test-slave-tcgx"
server_key: "b00a9d75e976d38a0da2cd79b66fcec6"
server_key_iv: "a042ee13d7a73e88c222f5c75a33c59d"
server_key_version: "encryptionkey@0"
I20260620 23:38:52.140497 13988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/minicluster-data/master-2/wal/instance:
uuid: "b25ed4cd67af4c2383c7bdda1640d0e4"
format_stamp: "Formatted at 2026-06-20 23:38:52 on dist-test-slave-tcgx"
server_key: "b00a9d75e976d38a0da2cd79b66fcec6"
server_key_iv: "a042ee13d7a73e88c222f5c75a33c59d"
server_key_version: "encryptionkey@0"
I20260620 23:38:52.146142 13988 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 23:38:52.150049 14003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:38:52.151700 13988 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 23:38:52.151875 13988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/minicluster-data/master-2/wal
uuid: "b25ed4cd67af4c2383c7bdda1640d0e4"
format_stamp: "Formatted at 2026-06-20 23:38:52 on dist-test-slave-tcgx"
server_key: "b00a9d75e976d38a0da2cd79b66fcec6"
server_key_iv: "a042ee13d7a73e88c222f5c75a33c59d"
server_key_version: "encryptionkey@0"
I20260620 23:38:52.152061 13988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:38:52.166682 13988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:38:52.172487 13988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:38:52.172986 13988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:38:52.194800 13988 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:40145
I20260620 23:38:52.194823 14054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:40145 every 8 connection(s)
I20260620 23:38:52.196312 13988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/minicluster-data/master-2/data/info.pb
I20260620 23:38:52.200431 14055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:38:52.204506 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 13988
I20260620 23:38:52.204713 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998525501163-388-0/minicluster-data/master-2/wal/instance
I20260620 23:38:52.205281 388 external_mini_cluster.cc:1428] Setting key 9a20b75fc35cf9a02788e7539c45e4ec
I20260620 23:38:52.208752 14055 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45075 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:38:52 dist-test-slave-tcgx krb5kdc[13827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 20 23:38:52 dist-test-slave-tcgx krb5kdc[13827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998732, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:38:52.227787 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:52.212639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38706 (local address 127.0.97.62:45075)
0620 23:38:52.212889 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 23:38:52.212897 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:38:52.213694 (+ 797us) server_negotiation.cc:407] Connection header received
0620 23:38:52.213894 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:38:52.213900 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:38:52.213982 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:38:52.214085 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:38:52.215268 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:52.215965 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:52.217411 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:52.217841 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:52.217986 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:52.221845 (+ 3859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:38:52.221850 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:38:52.221854 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:38:52.221891 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:38:52.224572 (+ 2681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:38:52.225348 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:38:52.225354 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:38:52.225357 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:38:52.225508 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:38:52.225946 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:38:52.225951 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:38:52.225954 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:38:52.226225 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:38:52.226402 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 23:38:52.227148 (+ 746us) server_negotiation.cc:299] Negotiation successful
0620 23:38:52.227402 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 23:38:52.228860 14059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:52.212631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45075 (local address 127.0.0.1:38706)
0620 23:38:52.213528 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0620 23:38:52.213576 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 23:38:52.213749 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:38:52.214147 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:38:52.214152 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:38:52.214766 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:38:52.215107 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:38:52.215117 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:38:52.216137 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:38:52.216142 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:38:52.217241 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:38:52.217254 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:38:52.218088 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:38:52.218094 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:38:52.218256 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:52.218909 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:38:52.218982 (+ 73us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:38:52.221622 (+ 2640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:38:52.224727 (+ 3105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:38:52.224741 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:38:52.224755 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:38:52.225208 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:38:52.225634 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:38:52.225639 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:38:52.225643 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:38:52.225809 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:38:52.226359 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:38:52.226367 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:38:52.226740 (+ 373us) client_negotiation.cc:769] Sending connection context
0620 23:38:52.227054 (+ 314us) client_negotiation.cc:240] Negotiation successful
0620 23:38:52.227586 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"spinlock_wait_cycles":16640,"thread_start_us":471,"threads_started":1}
I20260620 23:38:52.228883 14058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:52.207389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36062 (local address 127.0.97.60:40145)
0620 23:38:52.208161 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0620 23:38:52.208175 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 23:38:52.208218 (+ 43us) server_negotiation.cc:407] Connection header received
0620 23:38:52.209941 (+ 1723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:38:52.209967 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:38:52.210498 (+ 531us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:38:52.210990 (+ 492us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:38:52.212597 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:52.214390 (+ 1793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:52.215958 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:52.216705 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:52.216921 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:52.221575 (+ 4654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:38:52.221584 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:38:52.221598 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:38:52.221633 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:38:52.224725 (+ 3092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:38:52.225533 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:38:52.225547 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:38:52.225550 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:38:52.225708 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:38:52.226133 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:38:52.226143 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:38:52.226146 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:38:52.226662 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:38:52.227029 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0620 23:38:52.227385 (+ 356us) server_negotiation.cc:299] Negotiation successful
0620 23:38:52.227705 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":563,"thread_start_us":359,"threads_started":1}
I20260620 23:38:52.230695 14055 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44191 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:38:52 dist-test-slave-tcgx krb5kdc[13827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998732, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Jun 20 23:38:52 dist-test-slave-tcgx krb5kdc[13827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:38:52.246124 14059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:52.231681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44191 (local address 127.0.0.1:50794)
0620 23:38:52.231951 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 23:38:52.231968 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:38:52.232105 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:38:52.232529 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:38:52.232534 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:38:52.233056 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:38:52.233271 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:38:52.233283 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:38:52.234270 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:38:52.234275 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:38:52.235242 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:38:52.235254 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:38:52.235965 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:38:52.235971 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:38:52.236162 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:52.236774 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:38:52.236799 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:38:52.239441 (+ 2642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:38:52.243359 (+ 3918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:38:52.243365 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:38:52.243371 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:38:52.243816 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:38:52.244290 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:38:52.244296 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:38:52.244301 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:38:52.244503 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:38:52.245142 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:38:52.245146 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:38:52.245293 (+ 147us) client_negotiation.cc:769] Sending connection context
0620 23:38:52.245415 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 23:38:52.245773 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105}
I20260620 23:38:52.246465 13986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:52.231652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50794 (local address 127.0.97.61:44191)
0620 23:38:52.231879 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 23:38:52.231885 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:38:52.232038 (+ 153us) server_negotiation.cc:407] Connection header received
0620 23:38:52.232274 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:38:52.232281 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:38:52.232387 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:38:52.232508 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:38:52.233429 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:52.234115 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:52.235391 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:52.235810 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:52.235952 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:52.239690 (+ 3738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:38:52.239696 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:38:52.239699 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:38:52.239735 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:38:52.243193 (+ 3458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:38:52.243971 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:38:52.243977 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:38:52.243981 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:38:52.244162 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:38:52.244620 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:38:52.244628 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:38:52.244633 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:38:52.245010 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:38:52.245178 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 23:38:52.245619 (+ 441us) server_negotiation.cc:299] Negotiation successful
0620 23:38:52.245870 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 23:38:52.247529 14055 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40145 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:38:52.249490 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:52.232435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38716 (local address 127.0.97.62:45075)
0620 23:38:52.232779 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0620 23:38:52.232785 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:38:52.232804 (+ 19us) server_negotiation.cc:407] Connection header received
0620 23:38:52.232972 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:38:52.232978 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:38:52.233059 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:38:52.233165 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:38:52.234352 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:52.235194 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:52.236516 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:52.237203 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:52.237344 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:52.242504 (+ 5160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:38:52.242513 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:38:52.242521 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:38:52.242568 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:38:52.246244 (+ 3676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:38:52.247296 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:38:52.247301 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:38:52.247304 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:38:52.247502 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:38:52.247965 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:38:52.247970 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:38:52.247971 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:38:52.248319 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:38:52.248641 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0620 23:38:52.248907 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 23:38:52.249138 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204}
Jun 20 23:38:52 dist-test-slave-tcgx krb5kdc[13827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998732, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 20 23:38:52 dist-test-slave-tcgx krb5kdc[13827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 23:38:52.262420 14059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:52.248394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40145 (local address 127.0.0.1:36076)
0620 23:38:52.248569 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 23:38:52.248588 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:38:52.248689 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:38:52.249121 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:38:52.249125 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:38:52.249614 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:38:52.249786 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:38:52.249795 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:38:52.251074 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:38:52.251082 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:38:52.252516 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:38:52.252535 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:38:52.253232 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:38:52.253237 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:38:52.253456 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:52.254087 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:38:52.254103 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:38:52.256718 (+ 2615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:38:52.259891 (+ 3173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:38:52.259900 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:38:52.259907 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:38:52.260428 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:38:52.260843 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:38:52.260849 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:38:52.260853 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:38:52.261029 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:38:52.261545 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:38:52.261549 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:38:52.261700 (+ 151us) client_negotiation.cc:769] Sending connection context
0620 23:38:52.261840 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 23:38:52.262081 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 23:38:52.262691 14058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:52.248654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36076 (local address 127.0.97.60:40145)
0620 23:38:52.248827 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 23:38:52.248834 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:38:52.248850 (+ 16us) server_negotiation.cc:407] Connection header received
0620 23:38:52.248899 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:38:52.248904 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:38:52.248970 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:38:52.249063 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:38:52.249930 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:52.250969 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:52.252672 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:52.253083 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:52.253175 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:52.256929 (+ 3754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:38:52.256935 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:38:52.256944 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:38:52.256971 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:38:52.259697 (+ 2726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:38:52.260568 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:38:52.260574 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:38:52.260578 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:38:52.260720 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:38:52.261140 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:38:52.261144 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:38:52.261146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:38:52.261419 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:38:52.261567 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 23:38:52.261972 (+ 405us) server_negotiation.cc:299] Negotiation successful
0620 23:38:52.262258 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":44}
I20260620 23:38:52.267622 13986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:52.251328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50810 (local address 127.0.97.61:44191)
0620 23:38:52.251501 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 23:38:52.251505 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:38:52.251524 (+ 19us) server_negotiation.cc:407] Connection header received
0620 23:38:52.251696 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:38:52.251699 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:38:52.251754 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:38:52.251829 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:38:52.253366 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:52.254234 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:52.256076 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:52.256583 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:52.256746 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:52.261659 (+ 4913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:38:52.261665 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:38:52.261668 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:38:52.261702 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:38:52.264985 (+ 3283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:38:52.265840 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:38:52.265844 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:38:52.265846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:38:52.265993 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:38:52.266391 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:38:52.266394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:38:52.266396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:38:52.266637 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:38:52.266881 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0620 23:38:52.267229 (+ 348us) server_negotiation.cc:299] Negotiation successful
0620 23:38:52.267391 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 23:38:52.268924 14055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b25ed4cd67af4c2383c7bdda1640d0e4: Bootstrap starting.
I20260620 23:38:52.272994 14055 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b25ed4cd67af4c2383c7bdda1640d0e4: Neither blocks nor log segments found. Creating new log.
I20260620 23:38:52.273718 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:38:52.274190 14055 log.cc:824] T 00000000000000000000000000000000 P b25ed4cd67af4c2383c7bdda1640d0e4: Log is configured to *not* fsync() on all Append() calls
I20260620 23:38:52.277259 14055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b25ed4cd67af4c2383c7bdda1640d0e4: No bootstrap required, opened a new log
I20260620 23:38:52.291755 14055 raft_consensus.cc:348] T 00000000000000000000000000000000 P b25ed4cd67af4c2383c7bdda1640d0e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e0644fc188f48009d8b4189ba90cf15" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45075 } } peers { permanent_uuid: "a5b1f241ade64178938b982d7a9b524d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44191 } } peers { permanent_uuid: "b25ed4cd67af4c2383c7bdda1640d0e4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40145 } }
I20260620 23:38:52.292230 14055 raft_consensus.cc:374] T 00000000000000000000000000000000 P b25ed4cd67af4c2383c7bdda1640d0e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:38:52.292347 14055 raft_consensus.cc:729] T 00000000000000000000000000000000 P b25ed4cd67af4c2383c7bdda1640d0e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b25ed4cd67af4c2383c7bdda1640d0e4, State: Initialized, Role: FOLLOWER
I20260620 23:38:52.293205 14055 consensus_queue.cc:260] T 00000000000000000000000000000000 P b25ed4cd67af4c2383c7bdda1640d0e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e0644fc188f48009d8b4189ba90cf15" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45075 } } peers { permanent_uuid: "a5b1f241ade64178938b982d7a9b524d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44191 } } peers { permanent_uuid: "b25ed4cd67af4c2383c7bdda1640d0e4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40145 } }
I20260620 23:38:52.300422 14068 sys_catalog.cc:455] T 00000000000000000000000000000000 P b25ed4cd67af4c2383c7bdda1640d0e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e0644fc188f48009d8b4189ba90cf15" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45075 } } peers { permanent_uuid: "a5b1f241ade64178938b982d7a9b524d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44191 } } peers { permanent_uuid: "b25ed4cd67af4c2383c7bdda1640d0e4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40145 } } }
I20260620 23:38:52.300920 14068 sys_catalog.cc:458] T 00000000000000000000000000000000 P b25ed4cd67af4c2383c7bdda1640d0e4 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:38:52.303105 14055 sys_catalog.cc:564] T 00000000000000000000000000000000 P b25ed4cd67af4c2383c7bdda1640d0e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:38:52.305895 13986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:52.278732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50816 (local address 127.0.97.61:44191)
0620 23:38:52.279410 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0620 23:38:52.279416 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:38:52.287530 (+ 8114us) server_negotiation.cc:407] Connection header received
0620 23:38:52.287694 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:38:52.287703 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:38:52.287806 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:38:52.287926 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:38:52.290262 (+ 2336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:52.291395 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:52.293600 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:52.294261 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:52.294365 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:52.298505 (+ 4140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:38:52.298512 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:38:52.298519 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:38:52.298569 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:38:52.302293 (+ 3724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:38:52.303352 (+ 1059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:38:52.303358 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:38:52.303362 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:38:52.303520 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:38:52.304029 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:38:52.304035 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:38:52.304039 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:38:52.304442 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:38:52.304681 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0620 23:38:52.305245 (+ 564us) server_negotiation.cc:299] Negotiation successful
0620 23:38:52.305502 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528}
I20260620 23:38:52.307121 14058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:52.279217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36088 (local address 127.0.97.60:40145)
0620 23:38:52.279409 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 23:38:52.279415 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:38:52.286101 (+ 6686us) server_negotiation.cc:407] Connection header received
0620 23:38:52.286653 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:38:52.286663 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:38:52.286782 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:38:52.286859 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:38:52.289557 (+ 2698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:52.290699 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:52.292805 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:52.293538 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:52.293636 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:52.299196 (+ 5560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:38:52.299206 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:38:52.299214 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:38:52.299271 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:38:52.303097 (+ 3826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:38:52.303982 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:38:52.303987 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:38:52.303991 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:38:52.304148 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:38:52.304708 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:38:52.304716 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:38:52.304722 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:38:52.305239 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:38:52.305549 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0620 23:38:52.306226 (+ 677us) server_negotiation.cc:299] Negotiation successful
0620 23:38:52.306560 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 23:38:52.308173 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:52.278834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38732 (local address 127.0.97.62:45075)
0620 23:38:52.279052 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 23:38:52.279058 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:38:52.288281 (+ 9223us) server_negotiation.cc:407] Connection header received
0620 23:38:52.288435 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:38:52.288441 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:38:52.288531 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:38:52.288637 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:38:52.291139 (+ 2502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:52.292189 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:52.294680 (+ 2491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:52.295340 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:52.295448 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:52.299574 (+ 4126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:38:52.299581 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:38:52.299586 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:38:52.299632 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:38:52.304109 (+ 4477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:38:52.305071 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:38:52.305078 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:38:52.305081 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:38:52.305278 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:38:52.305720 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:38:52.305726 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:38:52.305730 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:38:52.306379 (+ 649us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:38:52.306615 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0620 23:38:52.307588 (+ 973us) server_negotiation.cc:299] Negotiation successful
0620 23:38:52.307827 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 23:38:52.310681 13907 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40145 } attempt: 4
Jun 20 23:38:52 dist-test-slave-tcgx krb5kdc[13827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998731, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
W20260620 23:38:52.326180 14079 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b25ed4cd67af4c2383c7bdda1640d0e4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:38:52.326320 14079 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:38:52.326908 13910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:52.311227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40145 (local address 127.0.0.1:36104)
0620 23:38:52.311411 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 23:38:52.311433 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 23:38:52.311557 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:38:52.312003 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:38:52.312008 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:38:52.312383 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:38:52.312528 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:38:52.312536 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:38:52.313807 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:38:52.313812 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:38:52.315232 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:38:52.315249 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:38:52.316040 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:38:52.316046 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:38:52.316233 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:52.316882 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:38:52.316907 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:38:52.319276 (+ 2369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:38:52.324073 (+ 4797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:38:52.324079 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:38:52.324083 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:38:52.324573 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:38:52.325062 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:38:52.325067 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:38:52.325070 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:38:52.325262 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:38:52.325987 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:38:52.325992 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:38:52.326174 (+ 182us) client_negotiation.cc:769] Sending connection context
0620 23:38:52.326307 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 23:38:52.326626 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 23:38:52.327723 14058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:52.311332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36104 (local address 127.0.97.60:40145)
0620 23:38:52.311533 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 23:38:52.311541 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:38:52.311566 (+ 25us) server_negotiation.cc:407] Connection header received
0620 23:38:52.311691 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:38:52.311701 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:38:52.311838 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:38:52.311979 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:38:52.312667 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:52.313663 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:52.315393 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:52.315884 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:52.316031 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:52.319466 (+ 3435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:38:52.319471 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:38:52.319476 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:38:52.319512 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:38:52.323903 (+ 4391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:38:52.324747 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:38:52.324756 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:38:52.324759 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:38:52.324915 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:38:52.325424 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:38:52.325432 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:38:52.325436 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:38:52.325837 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:38:52.327125 (+ 1288us) server_negotiation.cc:1035] Waiting for connection context
0620 23:38:52.327258 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 23:38:52.327464 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":55}
I20260620 23:38:52.321105 13988 master_runner.cc:186] Error getting master registration for 127.0.97.62:45075: OK, instance_id { permanent_uuid: "7e0644fc188f48009d8b4189ba90cf15" instance_seqno: 1781998731163073 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 23:38:52.332904 13988 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 23:38:52.333266 13907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e0644fc188f48009d8b4189ba90cf15: Bootstrap starting.
I20260620 23:38:52.336549 13907 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e0644fc188f48009d8b4189ba90cf15: Neither blocks nor log segments found. Creating new log.
I20260620 23:38:52.337618 13907 log.cc:824] T 00000000000000000000000000000000 P 7e0644fc188f48009d8b4189ba90cf15: Log is configured to *not* fsync() on all Append() calls
I20260620 23:38:52.340766 13907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e0644fc188f48009d8b4189ba90cf15: No bootstrap required, opened a new log
I20260620 23:38:52.349529 13907 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e0644fc188f48009d8b4189ba90cf15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e0644fc188f48009d8b4189ba90cf15" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45075 } } peers { permanent_uuid: "a5b1f241ade64178938b982d7a9b524d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44191 } } peers { permanent_uuid: "b25ed4cd67af4c2383c7bdda1640d0e4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40145 } }
I20260620 23:38:52.349996 13907 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e0644fc188f48009d8b4189ba90cf15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:38:52.350133 13907 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e0644fc188f48009d8b4189ba90cf15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e0644fc188f48009d8b4189ba90cf15, State: Initialized, Role: FOLLOWER
I20260620 23:38:52.350920 13907 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e0644fc188f48009d8b4189ba90cf15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e0644fc188f48009d8b4189ba90cf15" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45075 } } peers { permanent_uuid: "a5b1f241ade64178938b982d7a9b524d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44191 } } peers { permanent_uuid: "b25ed4cd67af4c2383c7bdda1640d0e4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40145 } }
I20260620 23:38:52.352005 14082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e0644fc188f48009d8b4189ba90cf15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e0644fc188f48009d8b4189ba90cf15" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45075 } } peers { permanent_uuid: "a5b1f241ade64178938b982d7a9b524d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44191 } } peers { permanent_uuid: "b25ed4cd67af4c2383c7bdda1640d0e4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40145 } } }
I20260620 23:38:52.352403 14082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e0644fc188f48009d8b4189ba90cf15 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:38:52.352970 14082 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e0644fc188f48009d8b4189ba90cf15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:38:52.353235 14082 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e0644fc188f48009d8b4189ba90cf15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e0644fc188f48009d8b4189ba90cf15" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45075 } } peers { permanent_uuid: "a5b1f241ade64178938b982d7a9b524d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44191 } } peers { permanent_uuid: "b25ed4cd67af4c2383c7bdda1640d0e4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40145 } }
I20260620 23:38:52.353543 13907 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e0644fc188f48009d8b4189ba90cf15 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:38:52.355883 14030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7e0644fc188f48009d8b4189ba90cf15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b25ed4cd67af4c2383c7bdda1640d0e4" is_pre_election: true
I20260620 23:38:52.356665 14030 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b25ed4cd67af4c2383c7bdda1640d0e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e0644fc188f48009d8b4189ba90cf15 in term 0.
I20260620 23:38:52.357182 14082 leader_election.cc:288] T 00000000000000000000000000000000 P 7e0644fc188f48009d8b4189ba90cf15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5b1f241ade64178938b982d7a9b524d (127.0.97.61:44191), b25ed4cd67af4c2383c7bdda1640d0e4 (127.0.97.60:40145)
I20260620 23:38:52.357448 13858 leader_election.cc:302] T 00000000000000000000000000000000 P 7e0644fc188f48009d8b4189ba90cf15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e0644fc188f48009d8b4189ba90cf15, b25ed4cd67af4c2383c7bdda1640d0e4; no voters:
I20260620 23:38:52.357993 14082 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e0644fc188f48009d8b4189ba90cf15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:38:52.358157 14082 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e0644fc188f48009d8b4189ba90cf15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:38:52.358273 14082 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e0644fc188f48009d8b4189ba90cf15 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:38:52.364784 14082 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e0644fc188f48009d8b4189ba90cf15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e0644fc188f48009d8b4189ba90cf15" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45075 } } peers { permanent_uuid: "a5b1f241ade64178938b982d7a9b524d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44191 } } peers { permanent_uuid: "b25ed4cd67af4c2383c7bdda1640d0e4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40145 } }
I20260620 23:38:52.365885 14082 leader_election.cc:288] T 00000000000000000000000000000000 P 7e0644fc188f48009d8b4189ba90cf15 [CANDIDATE]: Term 1 election: Requested vote from peers a5b1f241ade64178938b982d7a9b524d (127.0.97.61:44191), b25ed4cd67af4c2383c7bdda1640d0e4 (127.0.97.60:40145)
I20260620 23:38:52.366727 14030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7e0644fc188f48009d8b4189ba90cf15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b25ed4cd67af4c2383c7bdda1640d0e4"
I20260620 23:38:52.367007 14030 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b25ed4cd67af4c2383c7bdda1640d0e4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:38:52.370604 14030 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b25ed4cd67af4c2383c7bdda1640d0e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e0644fc188f48009d8b4189ba90cf15 in term 1.
I20260620 23:38:52.371320 13858 leader_election.cc:302] T 00000000000000000000000000000000 P 7e0644fc188f48009d8b4189ba90cf15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e0644fc188f48009d8b4189ba90cf15, b25ed4cd67af4c2383c7bdda1640d0e4; no voters:
I20260620 23:38:52.371688 14082 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e0644fc188f48009d8b4189ba90cf15 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:38:52.372346 14082 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e0644fc188f48009d8b4189ba90cf15 [term 1 LEADER]: Becoming Leader. State: Replica: 7e0644fc188f48009d8b4189ba90cf15, State: Running, Role: LEADER
I20260620 23:38:52.373140 14082 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e0644fc188f48009d8b4189ba90cf15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e0644fc188f48009d8b4189ba90cf15" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45075 } } peers { permanent_uuid: "a5b1f241ade64178938b982d7a9b524d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44191 } } peers { permanent_uuid: "b25ed4cd67af4c2383c7bdda1640d0e4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40145 } }
I20260620 23:38:52.380200 14090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e0644fc188f48009d8b4189ba90cf15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e0644fc188f48009d8b4189ba90cf15. Latest consensus state: current_term: 1 leader_uuid: "7e0644fc188f48009d8b4189ba90cf15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e0644fc188f48009d8b4189ba90cf15" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45075 } } peers { permanent_uuid: "a5b1f241ade64178938b982d7a9b524d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44191 } } peers { permanent_uuid: "b25ed4cd67af4c2383c7bdda1640d0e4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40145 } } }
I20260620 23:38:52.380913 14090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e0644fc188f48009d8b4189ba90cf15 [sys.catalog]: This master's current role is: LEADER
I20260620 23:38:52.382426 14094 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:38:52.392107 14094 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:38:52.386870 13840 master_runner.cc:186] Error getting master registration for 127.0.97.61:44191: OK, instance_id { permanent_uuid: "a5b1f241ade64178938b982d7a9b524d" instance_seqno: 1781998731686492 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 23:38:52.400918 13840 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 23:38:52.401516 13861 tablet.cc:2369] T 00000000000000000000000000000000 P 7e0644fc188f48009d8b4189ba90cf15: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:38:52.409873 14030 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b25ed4cd67af4c2383c7bdda1640d0e4 [term 1 FOLLOWER]: Refusing update from remote peer 7e0644fc188f48009d8b4189ba90cf15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:38:52.411729 14082 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7e0644fc188f48009d8b4189ba90cf15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b25ed4cd67af4c2383c7bdda1640d0e4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:38:52.418475 13981 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40145 } attempt: 5
W20260620 23:38:52.425922 14009 tablet.cc:2369] T 00000000000000000000000000000000 P b25ed4cd67af4c2383c7bdda1640d0e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 23:38:52 dist-test-slave-tcgx krb5kdc[13827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998731, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260620 23:38:52.433213 14068 sys_catalog.cc:455] T 00000000000000000000000000000000 P b25ed4cd67af4c2383c7bdda1640d0e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e0644fc188f48009d8b4189ba90cf15. Latest consensus state: current_term: 1 leader_uuid: "7e0644fc188f48009d8b4189ba90cf15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e0644fc188f48009d8b4189ba90cf15" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45075 } } peers { permanent_uuid: "a5b1f241ade64178938b982d7a9b524d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44191 } } peers { permanent_uuid: "b25ed4cd67af4c2383c7bdda1640d0e4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40145 } } }
I20260620 23:38:52.433576 14068 sys_catalog.cc:458] T 00000000000000000000000000000000 P b25ed4cd67af4c2383c7bdda1640d0e4 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:38:52.437178 13984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:52.419612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40145 (local address 127.0.0.1:36118)
0620 23:38:52.419874 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 23:38:52.419895 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:38:52.420028 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:38:52.420622 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:38:52.420628 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:38:52.421184 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:38:52.421424 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:38:52.421437 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:38:52.422963 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:38:52.422969 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:38:52.424509 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:38:52.424525 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:38:52.425708 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:38:52.425717 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:38:52.425919 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:52.426617 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:38:52.426641 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:38:52.429593 (+ 2952us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:38:52.434144 (+ 4551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:38:52.434150 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:38:52.434154 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:38:52.434709 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:38:52.435207 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:38:52.435213 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:38:52.435217 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:38:52.435396 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:38:52.436125 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:38:52.436130 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:38:52.436301 (+ 171us) client_negotiation.cc:769] Sending connection context
0620 23:38:52.436468 (+ 167us) client_negotiation.cc:240] Negotiation successful
0620 23:38:52.436768 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145}
I20260620 23:38:52.438177 14058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:52.419220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36118 (local address 127.0.97.60:40145)
0620 23:38:52.419421 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 23:38:52.419427 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:38:52.420136 (+ 709us) server_negotiation.cc:407] Connection header received
0620 23:38:52.420207 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:38:52.420212 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:38:52.420289 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:38:52.420396 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:38:52.421602 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:52.422743 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:52.424756 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:52.425370 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:52.425508 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:52.429836 (+ 4328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:38:52.429843 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:38:52.429849 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:38:52.429891 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:38:52.433947 (+ 4056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:38:52.434860 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:38:52.434866 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:38:52.434870 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:38:52.435070 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:38:52.435531 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:38:52.435537 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:38:52.435541 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:38:52.435897 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:38:52.436063 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 23:38:52.437563 (+ 1500us) server_negotiation.cc:299] Negotiation successful
0620 23:38:52.437798 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 23:38:52.444154 13981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5b1f241ade64178938b982d7a9b524d: Bootstrap starting.
I20260620 23:38:52.446604 14098 mvcc.cc:205] Tried to move back new op lower bound from 7299066807934148608 to 7299066807806177280. Current Snapshot: MvccSnapshot[applied={T|T < 7299066807934148608}]
I20260620 23:38:52.448787 13981 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5b1f241ade64178938b982d7a9b524d: Neither blocks nor log segments found. Creating new log.
I20260620 23:38:52.450219 13981 log.cc:824] T 00000000000000000000000000000000 P a5b1f241ade64178938b982d7a9b524d: Log is configured to *not* fsync() on all Append() calls
I20260620 23:38:52.460057 13981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5b1f241ade64178938b982d7a9b524d: No bootstrap required, opened a new log
I20260620 23:38:52.465205 14068 sys_catalog.cc:455] T 00000000000000000000000000000000 P b25ed4cd67af4c2383c7bdda1640d0e4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7e0644fc188f48009d8b4189ba90cf15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e0644fc188f48009d8b4189ba90cf15" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45075 } } peers { permanent_uuid: "a5b1f241ade64178938b982d7a9b524d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44191 } } peers { permanent_uuid: "b25ed4cd67af4c2383c7bdda1640d0e4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40145 } } }
I20260620 23:38:52.465487 14068 sys_catalog.cc:458] T 00000000000000000000000000000000 P b25ed4cd67af4c2383c7bdda1640d0e4 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:38:52.466012 14090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e0644fc188f48009d8b4189ba90cf15 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7e0644fc188f48009d8b4189ba90cf15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e0644fc188f48009d8b4189ba90cf15" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45075 } } peers { permanent_uuid: "a5b1f241ade64178938b982d7a9b524d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44191 } } peers { permanent_uuid: "b25ed4cd67af4c2383c7bdda1640d0e4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40145 } } }
I20260620 23:38:52.466328 14090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e0644fc188f48009d8b4189ba90cf15 [sys.catalog]: This master's current role is: LEADER
I20260620 23:38:52.471776 13981 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5b1f241ade64178938b982d7a9b524d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e0644fc188f48009d8b4189ba90cf15" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45075 } } peers { permanent_uuid: "a5b1f241ade64178938b982d7a9b524d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44191 } } peers { permanent_uuid: "b25ed4cd67af4c2383c7bdda1640d0e4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40145 } }
I20260620 23:38:52.472326 13981 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5b1f241ade64178938b982d7a9b524d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:38:52.472509 13981 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5b1f241ade64178938b982d7a9b524d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5b1f241ade64178938b982d7a9b524d, State: Initialized, Role: FOLLOWER
I20260620 23:38:52.473255 13981 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5b1f241ade64178938b982d7a9b524d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e0644fc188f48009d8b4189ba90cf15" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45075 } } peers { permanent_uuid: "a5b1f241ade64178938b982d7a9b524d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44191 } } peers { permanent_uuid: "b25ed4cd67af4c2383c7bdda1640d0e4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40145 } }
I20260620 23:38:52.476351 14094 catalog_manager.cc:1347] Generated new cluster ID: 726930ac9bdf4a5899183e6b7623e3fc
I20260620 23:38:52.476517 14094 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:38:52.480094 14104 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5b1f241ade64178938b982d7a9b524d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e0644fc188f48009d8b4189ba90cf15" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45075 } } peers { permanent_uuid: "a5b1f241ade64178938b982d7a9b524d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44191 } } peers { permanent_uuid: "b25ed4cd67af4c2383c7bdda1640d0e4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40145 } } }
I20260620 23:38:52.480849 14104 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5b1f241ade64178938b982d7a9b524d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:38:52.481567 13981 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5b1f241ade64178938b982d7a9b524d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:38:52.483047 13955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7e0644fc188f48009d8b4189ba90cf15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5b1f241ade64178938b982d7a9b524d"
I20260620 23:38:52.483559 13955 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5b1f241ade64178938b982d7a9b524d [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:38:52.483898 13956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7e0644fc188f48009d8b4189ba90cf15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5b1f241ade64178938b982d7a9b524d" is_pre_election: true
I20260620 23:38:52.487167 13955 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a5b1f241ade64178938b982d7a9b524d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e0644fc188f48009d8b4189ba90cf15 in term 1.
I20260620 23:38:52.487829 13954 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a5b1f241ade64178938b982d7a9b524d [term 1 FOLLOWER]: Refusing update from remote peer 7e0644fc188f48009d8b4189ba90cf15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:38:52.489512 14082 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7e0644fc188f48009d8b4189ba90cf15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5b1f241ade64178938b982d7a9b524d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:38:52.521822 14104 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5b1f241ade64178938b982d7a9b524d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e0644fc188f48009d8b4189ba90cf15. Latest consensus state: current_term: 1 leader_uuid: "7e0644fc188f48009d8b4189ba90cf15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e0644fc188f48009d8b4189ba90cf15" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45075 } } peers { permanent_uuid: "a5b1f241ade64178938b982d7a9b524d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44191 } } peers { permanent_uuid: "b25ed4cd67af4c2383c7bdda1640d0e4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40145 } } }
I20260620 23:38:52.522181 14104 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5b1f241ade64178938b982d7a9b524d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:38:52.522601 14094 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:38:52.524468 14094 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:38:52.525038 14112 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5b1f241ade64178938b982d7a9b524d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7e0644fc188f48009d8b4189ba90cf15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e0644fc188f48009d8b4189ba90cf15" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45075 } } peers { permanent_uuid: "a5b1f241ade64178938b982d7a9b524d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44191 } } peers { permanent_uuid: "b25ed4cd67af4c2383c7bdda1640d0e4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40145 } } }
I20260620 23:38:52.525617 14112 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5b1f241ade64178938b982d7a9b524d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:38:52.526759 14090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e0644fc188f48009d8b4189ba90cf15 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7e0644fc188f48009d8b4189ba90cf15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e0644fc188f48009d8b4189ba90cf15" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45075 } } peers { permanent_uuid: "a5b1f241ade64178938b982d7a9b524d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44191 } } peers { permanent_uuid: "b25ed4cd67af4c2383c7bdda1640d0e4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40145 } } }
I20260620 23:38:52.527024 14090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e0644fc188f48009d8b4189ba90cf15 [sys.catalog]: This master's current role is: LEADER
I20260620 23:38:52.545156 14094 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e0644fc188f48009d8b4189ba90cf15: Generated new TSK 0
I20260620 23:38:52.546406 14094 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:38:52.566006 14119 catalog_manager.cc:1259] Loaded cluster ID: 726930ac9bdf4a5899183e6b7623e3fc
I20260620 23:38:52.566237 14119 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a5b1f241ade64178938b982d7a9b524d: loading cluster ID for follower catalog manager: success
I20260620 23:38:52.570175 14119 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a5b1f241ade64178938b982d7a9b524d: acquiring CA information for follower catalog manager: success
I20260620 23:38:52.572858 14119 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a5b1f241ade64178938b982d7a9b524d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 23:38:53.328490 14079 catalog_manager.cc:1259] Loaded cluster ID: 726930ac9bdf4a5899183e6b7623e3fc
I20260620 23:38:53.328606 14079 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b25ed4cd67af4c2383c7bdda1640d0e4: loading cluster ID for follower catalog manager: success
I20260620 23:38:53.331306 14079 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b25ed4cd67af4c2383c7bdda1640d0e4: acquiring CA information for follower catalog manager: success
I20260620 23:38:53.333346 14079 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b25ed4cd67af4c2383c7bdda1640d0e4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 23:38:53.728268 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 13840
I20260620 23:38:53.729338 13902 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:38:53.840493 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 13840
I20260620 23:38:53.862727 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 13914
I20260620 23:38:53.863905 13976 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:38:53.972285 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 13914
I20260620 23:38:53.992554 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 13988
I20260620 23:38:53.993616 14050 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:38:54.100656 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 13988
2026-06-20T23:38:54Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4731 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781998525501163-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:38:54 dist-test-slave-tcgx krb5kdc[14132](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:38:54 dist-test-slave-tcgx krb5kdc[14132](info): set up 2 sockets
Jun 20 23:38:54 dist-test-slave-tcgx krb5kdc[14132](info): commencing operation
krb5kdc: starting...
W20260620 23:38:55.215579 388 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.006s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781998525501163-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781998525501163-388-0/krb5kdc/test-user.keytab.
Jun 20 23:38:55 dist-test-slave-tcgx krb5kdc[14132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998735, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:38:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:38:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 23:38:55.462210 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781998525501163-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781998525501163-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781998525501163-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45999
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45999
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:38:55.804190 14145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:38:55.804658 14145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:38:55.804843 14145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:38:55.812538 14145 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:38:55.812664 14145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:38:55.812747 14145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:38:55.812837 14145 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:38:55.812915 14145 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:38:55.823146 14145 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781998525501163-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781998525501163-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781998525501163-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:38:55.824591 14145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:38:55.825994 14145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:38:55.838234 14151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:38:55.838286 14150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:38:55.839120 14153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:38:55.839412 14145 server_base.cc:1034] running on GCE node
I20260620 23:38:55.840309 14145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:38:55.841928 14145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:38:55.843155 14145 hybrid_clock.cc:648] HybridClock initialized: now 1781998735843102 us; error 50 us; skew 500 ppm
I20260620 23:38:55.843513 14145 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:38:55 dist-test-slave-tcgx krb5kdc[14132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998735, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:38:55.846643 14145 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 23:38:55.848315 14145 webserver.cc:460] Webserver started at http://127.0.97.62:35695/ using document root <none> and password file <none>
I20260620 23:38:55.849072 14145 fs_manager.cc:359] Metadata directory not provided
I20260620 23:38:55.849224 14145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:38:55.849577 14145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:38:55.852715 14145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781998525501163-388-0/minicluster-data/master-0/data/instance:
uuid: "bd7e5be310b4459d8af687e60b05a570"
format_stamp: "Formatted at 2026-06-20 23:38:55 on dist-test-slave-tcgx"
server_key: "1e9f6e613651811ad0f6dace59e308a1"
server_key_iv: "ef0a3d9ff6de880cd22fc1e4fb158af7"
server_key_version: "encryptionkey@0"
I20260620 23:38:55.853507 14145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781998525501163-388-0/minicluster-data/master-0/wal/instance:
uuid: "bd7e5be310b4459d8af687e60b05a570"
format_stamp: "Formatted at 2026-06-20 23:38:55 on dist-test-slave-tcgx"
server_key: "1e9f6e613651811ad0f6dace59e308a1"
server_key_iv: "ef0a3d9ff6de880cd22fc1e4fb158af7"
server_key_version: "encryptionkey@0"
I20260620 23:38:55.859071 14145 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 23:38:55.862797 14160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:38:55.864315 14145 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:38:55.864529 14145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781998525501163-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781998525501163-388-0/minicluster-data/master-0/wal
uuid: "bd7e5be310b4459d8af687e60b05a570"
format_stamp: "Formatted at 2026-06-20 23:38:55 on dist-test-slave-tcgx"
server_key: "1e9f6e613651811ad0f6dace59e308a1"
server_key_iv: "ef0a3d9ff6de880cd22fc1e4fb158af7"
server_key_version: "encryptionkey@0"
I20260620 23:38:55.864761 14145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781998525501163-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781998525501163-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781998525501163-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:38:55.888552 14145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:38:55.893056 14145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:38:55.893361 14145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:38:55.912563 14145 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45999
I20260620 23:38:55.912604 14211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45999 every 8 connection(s)
I20260620 23:38:55.914028 14145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:38:55.916070 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 14145
I20260620 23:38:55.916298 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781998525501163-388-0/minicluster-data/master-0/wal/instance
I20260620 23:38:55.917155 388 external_mini_cluster.cc:1428] Setting key 34b5444b1c7bab30fadcf0e473c9228b
I20260620 23:38:55.918920 14212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:38:55.931028 14212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd7e5be310b4459d8af687e60b05a570: Bootstrap starting.
Jun 20 23:38:55 dist-test-slave-tcgx krb5kdc[14132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998735, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:38:55.934580 14212 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd7e5be310b4459d8af687e60b05a570: Neither blocks nor log segments found. Creating new log.
I20260620 23:38:55.935750 14212 log.cc:824] T 00000000000000000000000000000000 P bd7e5be310b4459d8af687e60b05a570: Log is configured to *not* fsync() on all Append() calls
I20260620 23:38:55.939235 14212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd7e5be310b4459d8af687e60b05a570: No bootstrap required, opened a new log
I20260620 23:38:55.940514 14215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:55.919192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46752 (local address 127.0.97.62:45999)
0620 23:38:55.919986 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0620 23:38:55.919998 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:38:55.920038 (+ 40us) server_negotiation.cc:407] Connection header received
0620 23:38:55.921699 (+ 1661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:38:55.921722 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:38:55.922157 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:38:55.922610 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:38:55.924225 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:55.925551 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:55.926892 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:55.927662 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:55.927897 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:55.932430 (+ 4533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:38:55.932442 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:38:55.932458 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:38:55.932511 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:38:55.935776 (+ 3265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:38:55.936783 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:38:55.936792 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:38:55.936805 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:38:55.937065 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:38:55.937518 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:38:55.937521 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:38:55.937523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:38:55.938080 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:38:55.938293 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 23:38:55.939074 (+ 781us) server_negotiation.cc:299] Negotiation successful
0620 23:38:55.939512 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":350,"threads_started":1}
I20260620 23:38:55.944885 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:38:55.949460 14212 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd7e5be310b4459d8af687e60b05a570 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd7e5be310b4459d8af687e60b05a570" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45999 } }
I20260620 23:38:55.949827 14212 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd7e5be310b4459d8af687e60b05a570 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:38:55.949899 14212 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd7e5be310b4459d8af687e60b05a570 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd7e5be310b4459d8af687e60b05a570, State: Initialized, Role: FOLLOWER
I20260620 23:38:55.950397 14212 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd7e5be310b4459d8af687e60b05a570 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd7e5be310b4459d8af687e60b05a570" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45999 } }
I20260620 23:38:55.950592 14212 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd7e5be310b4459d8af687e60b05a570 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:38:55.950687 14212 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd7e5be310b4459d8af687e60b05a570 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:38:55.950811 14212 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd7e5be310b4459d8af687e60b05a570 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:38:55.953012 14212 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd7e5be310b4459d8af687e60b05a570 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd7e5be310b4459d8af687e60b05a570" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45999 } }
I20260620 23:38:55.953533 14212 leader_election.cc:302] T 00000000000000000000000000000000 P bd7e5be310b4459d8af687e60b05a570 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd7e5be310b4459d8af687e60b05a570; no voters:
I20260620 23:38:55.954161 14212 leader_election.cc:288] T 00000000000000000000000000000000 P bd7e5be310b4459d8af687e60b05a570 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:38:55.954336 14217 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd7e5be310b4459d8af687e60b05a570 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:38:55.955168 14217 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd7e5be310b4459d8af687e60b05a570 [term 1 LEADER]: Becoming Leader. State: Replica: bd7e5be310b4459d8af687e60b05a570, State: Running, Role: LEADER
I20260620 23:38:55.955729 14217 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd7e5be310b4459d8af687e60b05a570 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd7e5be310b4459d8af687e60b05a570" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45999 } }
I20260620 23:38:55.956465 14212 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd7e5be310b4459d8af687e60b05a570 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:38:55.958634 14219 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd7e5be310b4459d8af687e60b05a570 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd7e5be310b4459d8af687e60b05a570. Latest consensus state: current_term: 1 leader_uuid: "bd7e5be310b4459d8af687e60b05a570" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd7e5be310b4459d8af687e60b05a570" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45999 } } }
I20260620 23:38:55.959064 14219 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd7e5be310b4459d8af687e60b05a570 [sys.catalog]: This master's current role is: LEADER
I20260620 23:38:55.958787 14218 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd7e5be310b4459d8af687e60b05a570 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd7e5be310b4459d8af687e60b05a570" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd7e5be310b4459d8af687e60b05a570" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45999 } } }
I20260620 23:38:55.959441 14218 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd7e5be310b4459d8af687e60b05a570 [sys.catalog]: This master's current role is: LEADER
I20260620 23:38:55.961146 14225 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:38:55.966665 14225 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:38:55.976014 14225 catalog_manager.cc:1347] Generated new cluster ID: c4e4a357c7c045908ea6e481d84adc2d
I20260620 23:38:55.976169 14225 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:38:56.005432 14225 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:38:56.006896 14225 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:38:56.019605 14225 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd7e5be310b4459d8af687e60b05a570: Generated new TSK 0
I20260620 23:38:56.020566 14225 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:38:56.058897 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 14145
I20260620 23:38:56.060264 14207 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:38:56.159303 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 14145
2026-06-20T23:38:56Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2054 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:38:56 dist-test-slave-tcgx krb5kdc[14240](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:38:56 dist-test-slave-tcgx krb5kdc[14240](info): set up 2 sockets
Jun 20 23:38:56 dist-test-slave-tcgx krb5kdc[14240](info): commencing operation
krb5kdc: starting...
W20260620 23:38:57.275449 388 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/krb5kdc/test-user.keytab.
Jun 20 23:38:57 dist-test-slave-tcgx krb5kdc[14240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:38:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:38:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 23:38:57.522106 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41617
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41617
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:38:57.864001 14253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:38:57.864465 14253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:38:57.864639 14253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:38:57.872071 14253 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:38:57.872254 14253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:38:57.872360 14253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:38:57.872463 14253 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:38:57.872526 14253 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:38:57.882649 14253 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41617
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:38:57.884160 14253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:38:57.885646 14253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:38:57.898025 14259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:38:57.898088 14261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:38:57.898063 14258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:38:57.898865 14253 server_base.cc:1034] running on GCE node
I20260620 23:38:57.899714 14253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:38:57.901260 14253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:38:57.902511 14253 hybrid_clock.cc:648] HybridClock initialized: now 1781998737902470 us; error 60 us; skew 500 ppm
I20260620 23:38:57.902873 14253 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:38:57 dist-test-slave-tcgx krb5kdc[14240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998737, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:38:57.906050 14253 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 23:38:57.907712 14253 webserver.cc:460] Webserver started at http://127.0.97.62:44471/ using document root <none> and password file <none>
I20260620 23:38:57.908471 14253 fs_manager.cc:359] Metadata directory not provided
I20260620 23:38:57.908632 14253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:38:57.908964 14253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:38:57.912298 14253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/master-0/data/instance:
uuid: "f7e7ba8cc23b4fdbaf0a02f1a75e37b9"
format_stamp: "Formatted at 2026-06-20 23:38:57 on dist-test-slave-tcgx"
server_key: "b3b53e6a0c350ff5e38b66f00ea6834e"
server_key_iv: "2503cbea121e0a69915fc15e3a7ce45b"
server_key_version: "encryptionkey@0"
I20260620 23:38:57.913172 14253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/master-0/wal/instance:
uuid: "f7e7ba8cc23b4fdbaf0a02f1a75e37b9"
format_stamp: "Formatted at 2026-06-20 23:38:57 on dist-test-slave-tcgx"
server_key: "b3b53e6a0c350ff5e38b66f00ea6834e"
server_key_iv: "2503cbea121e0a69915fc15e3a7ce45b"
server_key_version: "encryptionkey@0"
I20260620 23:38:57.919234 14253 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260620 23:38:57.923175 14268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:38:57.924746 14253 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:38:57.925014 14253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/master-0/wal
uuid: "f7e7ba8cc23b4fdbaf0a02f1a75e37b9"
format_stamp: "Formatted at 2026-06-20 23:38:57 on dist-test-slave-tcgx"
server_key: "b3b53e6a0c350ff5e38b66f00ea6834e"
server_key_iv: "2503cbea121e0a69915fc15e3a7ce45b"
server_key_version: "encryptionkey@0"
I20260620 23:38:57.925268 14253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:38:57.947132 14253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:38:57.951388 14253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:38:57.951714 14253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:38:57.971362 14253 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41617
I20260620 23:38:57.971411 14319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41617 every 8 connection(s)
I20260620 23:38:57.972929 14253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:38:57.976408 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 14253
I20260620 23:38:57.976634 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/master-0/wal/instance
I20260620 23:38:57.977205 14320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:38:57.977349 388 external_mini_cluster.cc:1428] Setting key 999f1440261f25dfc9a14cda248ca964
I20260620 23:38:57.992857 14320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7e7ba8cc23b4fdbaf0a02f1a75e37b9: Bootstrap starting.
Jun 20 23:38:57 dist-test-slave-tcgx krb5kdc[14240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:38:57.997207 14320 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7e7ba8cc23b4fdbaf0a02f1a75e37b9: Neither blocks nor log segments found. Creating new log.
I20260620 23:38:57.998701 14320 log.cc:824] T 00000000000000000000000000000000 P f7e7ba8cc23b4fdbaf0a02f1a75e37b9: Log is configured to *not* fsync() on all Append() calls
I20260620 23:38:58.002638 14320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7e7ba8cc23b4fdbaf0a02f1a75e37b9: No bootstrap required, opened a new log
I20260620 23:38:58.004659 14323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:57.979556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36748 (local address 127.0.97.62:41617)
0620 23:38:57.980487 (+ 931us) server_negotiation.cc:206] Beginning negotiation
0620 23:38:57.980503 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 23:38:57.980564 (+ 61us) server_negotiation.cc:407] Connection header received
0620 23:38:57.982647 (+ 2083us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:38:57.982679 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:38:57.983308 (+ 629us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:38:57.983804 (+ 496us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:38:57.985167 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:57.987324 (+ 2157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:57.988733 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:57.989932 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:57.990169 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:57.994627 (+ 4458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:38:57.994641 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:38:57.994663 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:38:57.994733 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:38:57.999089 (+ 4356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:38:58.000193 (+ 1104us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:38:58.000200 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:38:58.000207 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:38:58.000500 (+ 293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:38:58.001055 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:38:58.001060 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:38:58.001063 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:38:58.001819 (+ 756us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:38:58.002129 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0620 23:38:58.002597 (+ 468us) server_negotiation.cc:299] Negotiation successful
0620 23:38:58.003121 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":422,"threads_started":1}
I20260620 23:38:58.012136 14320 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7e7ba8cc23b4fdbaf0a02f1a75e37b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7e7ba8cc23b4fdbaf0a02f1a75e37b9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41617 } }
I20260620 23:38:58.012688 14320 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7e7ba8cc23b4fdbaf0a02f1a75e37b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:38:58.012843 14320 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7e7ba8cc23b4fdbaf0a02f1a75e37b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7e7ba8cc23b4fdbaf0a02f1a75e37b9, State: Initialized, Role: FOLLOWER
I20260620 23:38:58.013480 14320 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7e7ba8cc23b4fdbaf0a02f1a75e37b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7e7ba8cc23b4fdbaf0a02f1a75e37b9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41617 } }
I20260620 23:38:58.013741 14320 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7e7ba8cc23b4fdbaf0a02f1a75e37b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:38:58.013914 14320 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7e7ba8cc23b4fdbaf0a02f1a75e37b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:38:58.014096 14320 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7e7ba8cc23b4fdbaf0a02f1a75e37b9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:38:58.016485 14320 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7e7ba8cc23b4fdbaf0a02f1a75e37b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7e7ba8cc23b4fdbaf0a02f1a75e37b9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41617 } }
I20260620 23:38:58.017138 14320 leader_election.cc:302] T 00000000000000000000000000000000 P f7e7ba8cc23b4fdbaf0a02f1a75e37b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7e7ba8cc23b4fdbaf0a02f1a75e37b9; no voters:
I20260620 23:38:58.017887 14320 leader_election.cc:288] T 00000000000000000000000000000000 P f7e7ba8cc23b4fdbaf0a02f1a75e37b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:38:58.018054 14325 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7e7ba8cc23b4fdbaf0a02f1a75e37b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:38:58.018765 14325 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7e7ba8cc23b4fdbaf0a02f1a75e37b9 [term 1 LEADER]: Becoming Leader. State: Replica: f7e7ba8cc23b4fdbaf0a02f1a75e37b9, State: Running, Role: LEADER
I20260620 23:38:58.019381 14325 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7e7ba8cc23b4fdbaf0a02f1a75e37b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7e7ba8cc23b4fdbaf0a02f1a75e37b9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41617 } }
I20260620 23:38:58.020190 14320 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7e7ba8cc23b4fdbaf0a02f1a75e37b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:38:58.022871 14327 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7e7ba8cc23b4fdbaf0a02f1a75e37b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7e7ba8cc23b4fdbaf0a02f1a75e37b9. Latest consensus state: current_term: 1 leader_uuid: "f7e7ba8cc23b4fdbaf0a02f1a75e37b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7e7ba8cc23b4fdbaf0a02f1a75e37b9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41617 } } }
I20260620 23:38:58.023612 14327 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7e7ba8cc23b4fdbaf0a02f1a75e37b9 [sys.catalog]: This master's current role is: LEADER
I20260620 23:38:58.022589 14326 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7e7ba8cc23b4fdbaf0a02f1a75e37b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7e7ba8cc23b4fdbaf0a02f1a75e37b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7e7ba8cc23b4fdbaf0a02f1a75e37b9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41617 } } }
I20260620 23:38:58.024055 14326 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7e7ba8cc23b4fdbaf0a02f1a75e37b9 [sys.catalog]: This master's current role is: LEADER
I20260620 23:38:58.025856 14334 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:38:58.030790 14334 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:38:58.040201 14334 catalog_manager.cc:1347] Generated new cluster ID: 7523bac8126c481e9cfe3986e84b2d5f
I20260620 23:38:58.040391 14334 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:38:58.061008 14334 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:38:58.062549 14334 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:38:58.071460 14334 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7e7ba8cc23b4fdbaf0a02f1a75e37b9: Generated new TSK 0
I20260620 23:38:58.072386 14334 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 23:38:58.139752 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41617
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:38:58.487115 14346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:38:58.487502 14346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:38:58.487685 14346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:38:58.495276 14346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:38:58.495482 14346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:38:58.495699 14346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 23:38:58.506376 14346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:38:58.508008 14346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:38:58.509424 14346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:38:58.523969 14351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:38:58.523993 14352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:38:58.524168 14354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:38:58.524969 14346 server_base.cc:1034] running on GCE node
I20260620 23:38:58.525662 14346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:38:58.526721 14346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:38:58.527976 14346 hybrid_clock.cc:648] HybridClock initialized: now 1781998738527858 us; error 79 us; skew 500 ppm
I20260620 23:38:58.528321 14346 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:38:58 dist-test-slave-tcgx krb5kdc[14240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998738, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:38:58.531852 14346 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 23:38:58.533529 14346 webserver.cc:460] Webserver started at http://127.0.97.1:34901/ using document root <none> and password file <none>
I20260620 23:38:58.534292 14346 fs_manager.cc:359] Metadata directory not provided
I20260620 23:38:58.534433 14346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:38:58.534772 14346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:38:58.537883 14346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-0/data/instance:
uuid: "a55a5d4f8a194f52b6da9351fdc8bdb1"
format_stamp: "Formatted at 2026-06-20 23:38:58 on dist-test-slave-tcgx"
server_key: "1c58a1c8ed30db672d5e9ccb3c142f8e"
server_key_iv: "b1cbda2f23d38799a6c9e3eacaa16b46"
server_key_version: "encryptionkey@0"
I20260620 23:38:58.538692 14346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-0/wal/instance:
uuid: "a55a5d4f8a194f52b6da9351fdc8bdb1"
format_stamp: "Formatted at 2026-06-20 23:38:58 on dist-test-slave-tcgx"
server_key: "1c58a1c8ed30db672d5e9ccb3c142f8e"
server_key_iv: "b1cbda2f23d38799a6c9e3eacaa16b46"
server_key_version: "encryptionkey@0"
I20260620 23:38:58.544060 14346 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 23:38:58.547854 14361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:38:58.549628 14346 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:38:58.549811 14346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-0/wal
uuid: "a55a5d4f8a194f52b6da9351fdc8bdb1"
format_stamp: "Formatted at 2026-06-20 23:38:58 on dist-test-slave-tcgx"
server_key: "1c58a1c8ed30db672d5e9ccb3c142f8e"
server_key_iv: "b1cbda2f23d38799a6c9e3eacaa16b46"
server_key_version: "encryptionkey@0"
I20260620 23:38:58.550031 14346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:38:58.572841 14346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:38:58.577450 14346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:38:58.577770 14346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:38:58.578796 14346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:38:58.580224 14346 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:38:58.580302 14346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:38:58.580451 14346 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:38:58.580515 14346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:38:58.614786 14346 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45841
I20260620 23:38:58.614925 14473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45841 every 8 connection(s)
I20260620 23:38:58.616211 14346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
I20260620 23:38:58.616856 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 14346
I20260620 23:38:58.617079 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-0/wal/instance
I20260620 23:38:58.617975 388 external_mini_cluster.cc:1428] Setting key 36728be2c71af14d0774b6e1163e05a4
Jun 20 23:38:58 dist-test-slave-tcgx krb5kdc[14240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998738, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:38:58.640062 14476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:58.619311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40801 (local address 127.0.97.62:41617)
0620 23:38:58.619779 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0620 23:38:58.619784 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:38:58.621270 (+ 1486us) server_negotiation.cc:407] Connection header received
0620 23:38:58.623271 (+ 2001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:38:58.623280 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:38:58.623399 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:38:58.623539 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:38:58.625297 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:58.626453 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:58.628160 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:58.628910 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:58.629028 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:58.633523 (+ 4495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:38:58.633529 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:38:58.633532 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:38:58.633570 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:38:58.636529 (+ 2959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:38:58.637383 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:38:58.637394 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:38:58.637397 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:38:58.637520 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:38:58.638100 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:38:58.638111 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:38:58.638113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:38:58.638388 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:38:58.638556 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 23:38:58.639415 (+ 859us) server_negotiation.cc:299] Negotiation successful
0620 23:38:58.639661 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":261,"threads_started":1}
I20260620 23:38:58.641106 14477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:58.620128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41617 (local address 127.0.97.1:40801)
0620 23:38:58.621109 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0620 23:38:58.621158 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 23:38:58.622907 (+ 1749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:38:58.623761 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:38:58.623770 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:38:58.624410 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:38:58.625106 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:38:58.625125 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:38:58.626620 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:38:58.626624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:38:58.627993 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:38:58.628006 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:38:58.629161 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:38:58.629170 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:38:58.629519 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:58.630333 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:38:58.630369 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:38:58.633258 (+ 2889us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:38:58.636711 (+ 3453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:38:58.636722 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:38:58.636741 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:38:58.637221 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:38:58.637645 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:38:58.637650 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:38:58.637653 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:38:58.637957 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:38:58.638527 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:38:58.638535 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:38:58.638980 (+ 445us) client_negotiation.cc:769] Sending connection context
0620 23:38:58.639304 (+ 324us) client_negotiation.cc:240] Negotiation successful
0620 23:38:58.639891 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"thread_start_us":440,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 23:38:58.642916 14474 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41617
I20260620 23:38:58.643297 14474 heartbeater.cc:464] Registering TS with master...
I20260620 23:38:58.644126 14474 heartbeater.cc:511] Master 127.0.97.62:41617 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260620 23:38:58.646622 14285 ts_manager.cc:194] Registered new tserver with Master: a55a5d4f8a194f52b6da9351fdc8bdb1 (127.0.97.1:45841)
I20260620 23:38:58.648211 14285 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:40801
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 23:38:58.673820 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41617
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:38:59.012789 14480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:38:59.013104 14480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:38:59.013201 14480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:38:59.020845 14480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:38:59.020967 14480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:38:59.021082 14480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 23:38:59.031741 14480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:38:59.033032 14480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:38:59.034336 14480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:38:59.048452 14485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:38:59.048375 14488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:38:59.048417 14486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:38:59.049387 14480 server_base.cc:1034] running on GCE node
I20260620 23:38:59.050035 14480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:38:59.051103 14480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:38:59.052338 14480 hybrid_clock.cc:648] HybridClock initialized: now 1781998739052282 us; error 51 us; skew 500 ppm
I20260620 23:38:59.052683 14480 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:38:59 dist-test-slave-tcgx krb5kdc[14240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998739, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:38:59.056183 14480 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 23:38:59.057662 14480 webserver.cc:460] Webserver started at http://127.0.97.2:40057/ using document root <none> and password file <none>
I20260620 23:38:59.058373 14480 fs_manager.cc:359] Metadata directory not provided
I20260620 23:38:59.058514 14480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:38:59.058825 14480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:38:59.061870 14480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-1/data/instance:
uuid: "acc9e1859c3e45aa9ab8428c27657281"
format_stamp: "Formatted at 2026-06-20 23:38:59 on dist-test-slave-tcgx"
server_key: "b4b7aa72b22d1327b1c18c01f62a07b3"
server_key_iv: "faf10a0724877051055956a791c3da90"
server_key_version: "encryptionkey@0"
I20260620 23:38:59.062709 14480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-1/wal/instance:
uuid: "acc9e1859c3e45aa9ab8428c27657281"
format_stamp: "Formatted at 2026-06-20 23:38:59 on dist-test-slave-tcgx"
server_key: "b4b7aa72b22d1327b1c18c01f62a07b3"
server_key_iv: "faf10a0724877051055956a791c3da90"
server_key_version: "encryptionkey@0"
I20260620 23:38:59.068186 14480 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 23:38:59.071861 14495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:38:59.073215 14480 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:38:59.073396 14480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-1/wal
uuid: "acc9e1859c3e45aa9ab8428c27657281"
format_stamp: "Formatted at 2026-06-20 23:38:59 on dist-test-slave-tcgx"
server_key: "b4b7aa72b22d1327b1c18c01f62a07b3"
server_key_iv: "faf10a0724877051055956a791c3da90"
server_key_version: "encryptionkey@0"
I20260620 23:38:59.073606 14480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:38:59.091150 14480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:38:59.095507 14480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:38:59.095801 14480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:38:59.096799 14480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:38:59.098227 14480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:38:59.098300 14480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:38:59.098387 14480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:38:59.098431 14480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:38:59.133842 14480 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39353
I20260620 23:38:59.134009 14607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39353 every 8 connection(s)
I20260620 23:38:59.135246 14480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
I20260620 23:38:59.138230 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 14480
I20260620 23:38:59.138443 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-1/wal/instance
I20260620 23:38:59.138912 388 external_mini_cluster.cc:1428] Setting key 9e9d80589807390d9beba62bdc002d99
Jun 20 23:38:59 dist-test-slave-tcgx krb5kdc[14240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998739, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:38:59.156715 14476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:59.138067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40165 (local address 127.0.97.62:41617)
0620 23:38:59.138309 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 23:38:59.138318 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:38:59.139641 (+ 1323us) server_negotiation.cc:407] Connection header received
0620 23:38:59.141497 (+ 1856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:38:59.141504 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:38:59.141595 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:38:59.141706 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:38:59.143337 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:59.144483 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:59.146066 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:59.146842 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:59.147014 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:59.150530 (+ 3516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:38:59.150536 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:38:59.150539 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:38:59.150575 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:38:59.153312 (+ 2737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:38:59.154242 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:38:59.154248 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:38:59.154252 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:38:59.154379 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:38:59.154844 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:38:59.154896 (+ 52us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:38:59.154899 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:38:59.155222 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:38:59.155388 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 23:38:59.156110 (+ 722us) server_negotiation.cc:299] Negotiation successful
0620 23:38:59.156349 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260620 23:38:59.157830 14610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:59.138584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41617 (local address 127.0.97.2:40165)
0620 23:38:59.139444 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0620 23:38:59.139503 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 23:38:59.141218 (+ 1715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:38:59.141984 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:38:59.141993 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:38:59.142507 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:38:59.143152 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:38:59.143171 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:38:59.144648 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:38:59.144652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:38:59.145902 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:38:59.145914 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:38:59.147123 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:38:59.147129 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:38:59.147326 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:59.148046 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:38:59.148074 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:38:59.150258 (+ 2184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:38:59.153475 (+ 3217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:38:59.153484 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:38:59.153502 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:38:59.154107 (+ 605us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:38:59.154499 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:38:59.154505 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:38:59.154509 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:38:59.154717 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:38:59.155362 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:38:59.155370 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:38:59.155740 (+ 370us) client_negotiation.cc:769] Sending connection context
0620 23:38:59.156021 (+ 281us) client_negotiation.cc:240] Negotiation successful
0620 23:38:59.156589 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":382,"threads_started":1}
I20260620 23:38:59.159529 14608 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41617
I20260620 23:38:59.159880 14608 heartbeater.cc:464] Registering TS with master...
I20260620 23:38:59.160656 14608 heartbeater.cc:511] Master 127.0.97.62:41617 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 23:38:59.162283 14285 ts_manager.cc:194] Registered new tserver with Master: acc9e1859c3e45aa9ab8428c27657281 (127.0.97.2:39353)
I20260620 23:38:59.163194 14285 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:40165
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 23:38:59.194730 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41617
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:38:59.540716 14613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:38:59.541033 14613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:38:59.541198 14613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:38:59.548987 14613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:38:59.549122 14613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:38:59.549336 14613 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 23:38:59.559741 14613 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:38:59.561159 14613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:38:59.562505 14613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:38:59.576897 14618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:38:59.576946 14619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:38:59.577497 14621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:38:59.577790 14613 server_base.cc:1034] running on GCE node
I20260620 23:38:59.578433 14613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:38:59.579504 14613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:38:59.580755 14613 hybrid_clock.cc:648] HybridClock initialized: now 1781998739580706 us; error 63 us; skew 500 ppm
I20260620 23:38:59.581136 14613 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:38:59 dist-test-slave-tcgx krb5kdc[14240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998739, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:38:59.584575 14613 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 23:38:59.586267 14613 webserver.cc:460] Webserver started at http://127.0.97.3:36397/ using document root <none> and password file <none>
I20260620 23:38:59.587110 14613 fs_manager.cc:359] Metadata directory not provided
I20260620 23:38:59.587271 14613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:38:59.587602 14613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:38:59.590667 14613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-2/data/instance:
uuid: "8a1301fde2de4afe8548772edc8267a9"
format_stamp: "Formatted at 2026-06-20 23:38:59 on dist-test-slave-tcgx"
server_key: "fb6b647b34e634d0ad7fd493cb6f7ed4"
server_key_iv: "0272705e08f39d14596d3236fc873eca"
server_key_version: "encryptionkey@0"
I20260620 23:38:59.591626 14613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-2/wal/instance:
uuid: "8a1301fde2de4afe8548772edc8267a9"
format_stamp: "Formatted at 2026-06-20 23:38:59 on dist-test-slave-tcgx"
server_key: "fb6b647b34e634d0ad7fd493cb6f7ed4"
server_key_iv: "0272705e08f39d14596d3236fc873eca"
server_key_version: "encryptionkey@0"
I20260620 23:38:59.597266 14613 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260620 23:38:59.601595 14628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:38:59.603201 14613 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:38:59.603412 14613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-2/wal
uuid: "8a1301fde2de4afe8548772edc8267a9"
format_stamp: "Formatted at 2026-06-20 23:38:59 on dist-test-slave-tcgx"
server_key: "fb6b647b34e634d0ad7fd493cb6f7ed4"
server_key_iv: "0272705e08f39d14596d3236fc873eca"
server_key_version: "encryptionkey@0"
I20260620 23:38:59.603649 14613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:38:59.618723 14613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:38:59.623268 14613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:38:59.623595 14613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:38:59.624737 14613 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:38:59.626157 14613 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:38:59.626271 14613 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:38:59.626443 14613 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:38:59.626528 14613 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:38:59.651432 14474 heartbeater.cc:503] Master 127.0.97.62:41617 was elected leader, sending a full tablet report...
I20260620 23:38:59.663434 14613 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41913
I20260620 23:38:59.663492 14740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41913 every 8 connection(s)
I20260620 23:38:59.664793 14613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
I20260620 23:38:59.669551 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 14613
I20260620 23:38:59.669756 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-2/wal/instance
I20260620 23:38:59.670312 388 external_mini_cluster.cc:1428] Setting key d1414e511ecc1efa8755feb9e14554fe
Jun 20 23:38:59 dist-test-slave-tcgx krb5kdc[14240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998739, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:38:59.686487 14743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:59.667641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48667 (local address 127.0.97.62:41617)
0620 23:38:59.668129 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0620 23:38:59.668134 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:38:59.669217 (+ 1083us) server_negotiation.cc:407] Connection header received
0620 23:38:59.671674 (+ 2457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:38:59.671679 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:38:59.671774 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:38:59.671921 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:38:59.673478 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:59.674282 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:59.675983 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:59.676397 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:59.676539 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:59.680302 (+ 3763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:38:59.680308 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:38:59.680311 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:38:59.680348 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:38:59.683282 (+ 2934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:38:59.684048 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:38:59.684053 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:38:59.684055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:38:59.684182 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:38:59.684642 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:38:59.684652 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:38:59.684654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:38:59.684934 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:38:59.685110 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 23:38:59.685849 (+ 739us) server_negotiation.cc:299] Negotiation successful
0620 23:38:59.686086 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":252,"threads_started":1}
I20260620 23:38:59.687621 14744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:59.668233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41617 (local address 127.0.97.3:48667)
0620 23:38:59.669000 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0620 23:38:59.669039 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 23:38:59.671359 (+ 2320us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:38:59.672178 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:38:59.672188 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:38:59.672719 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:38:59.673300 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:38:59.673316 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:38:59.674472 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:38:59.674476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:38:59.675830 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:38:59.675843 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:38:59.676635 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:38:59.676642 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:38:59.676880 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:59.677665 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:38:59.677687 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:38:59.680061 (+ 2374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:38:59.683425 (+ 3364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:38:59.683433 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:38:59.683447 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:38:59.683888 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:38:59.684301 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:38:59.684307 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:38:59.684310 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:38:59.684512 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:38:59.685079 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:38:59.685088 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:38:59.685457 (+ 369us) client_negotiation.cc:769] Sending connection context
0620 23:38:59.685740 (+ 283us) client_negotiation.cc:240] Negotiation successful
0620 23:38:59.686338 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":309,"threads_started":1}
I20260620 23:38:59.689036 14741 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41617
I20260620 23:38:59.689316 14741 heartbeater.cc:464] Registering TS with master...
I20260620 23:38:59.690011 14741 heartbeater.cc:511] Master 127.0.97.62:41617 requested a full tablet report, sending...
I20260620 23:38:59.691437 14285 ts_manager.cc:194] Registered new tserver with Master: 8a1301fde2de4afe8548772edc8267a9 (127.0.97.3:41913)
I20260620 23:38:59.692216 14285 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:48667
I20260620 23:38:59.698596 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:38:59.716204 14743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:59.703263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36764 (local address 127.0.97.62:41617)
0620 23:38:59.703445 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 23:38:59.703449 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:38:59.703896 (+ 447us) server_negotiation.cc:407] Connection header received
0620 23:38:59.703991 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:38:59.703995 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:38:59.704062 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:38:59.704151 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:38:59.705501 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:59.706259 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:59.707901 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:59.708430 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:59.708533 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:59.711099 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:38:59.711103 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:38:59.711107 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:38:59.711140 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:38:59.713747 (+ 2607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:38:59.714441 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:38:59.714445 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:38:59.714447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:38:59.714574 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:38:59.714903 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:38:59.714906 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:38:59.714908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:38:59.715319 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:38:59.715512 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 23:38:59.715795 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 23:38:59.715972 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 23:38:59.720882 14285 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36764:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:38:59.728335 14285 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:38:59.763417 14757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:59.742061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46004 (local address 127.0.97.2:39353)
0620 23:38:59.755274 (+ 13213us) server_negotiation.cc:206] Beginning negotiation
0620 23:38:59.755280 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:38:59.755304 (+ 24us) server_negotiation.cc:407] Connection header received
0620 23:38:59.755377 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:38:59.755382 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:38:59.755583 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:38:59.756530 (+ 947us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:38:59.756708 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:59.758500 (+ 1792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:59.761146 (+ 2646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:59.762138 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:59.762300 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:59.762530 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0620 23:38:59.762870 (+ 340us) server_negotiation.cc:299] Negotiation successful
0620 23:38:59.763072 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13074,"thread_start_us":12960,"threads_started":1}
I20260620 23:38:59.763535 14756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:59.741306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39353 (local address 127.0.0.1:46004)
0620 23:38:59.752696 (+ 11390us) negotiation.cc:107] Waiting for socket to connect
0620 23:38:59.752750 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 23:38:59.752965 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:38:59.755812 (+ 2847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:38:59.755816 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:38:59.755840 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:38:59.756341 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:38:59.756355 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:38:59.758647 (+ 2292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:38:59.758652 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:38:59.760999 (+ 2347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:38:59.761010 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:38:59.762359 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:38:59.762365 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:38:59.762572 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:59.762636 (+ 64us) client_negotiation.cc:769] Sending connection context
0620 23:38:59.762803 (+ 167us) client_negotiation.cc:240] Negotiation successful
0620 23:38:59.763043 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11087,"mutex_wait_us":3,"thread_start_us":12798,"threads_started":1}
I20260620 23:38:59.763684 14755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:59.741828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41913 (local address 127.0.0.1:33238)
0620 23:38:59.754766 (+ 12938us) negotiation.cc:107] Waiting for socket to connect
0620 23:38:59.754784 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:38:59.754914 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:38:59.755646 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:38:59.755653 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:38:59.755687 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:38:59.756215 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:38:59.756230 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:38:59.757809 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:38:59.757816 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:38:59.760706 (+ 2890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:38:59.760720 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:38:59.762730 (+ 2010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:38:59.762734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:38:59.762988 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:59.763024 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 23:38:59.763110 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 23:38:59.763252 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12805,"mutex_wait_us":193,"thread_start_us":11380,"threads_started":1}
I20260620 23:38:59.764416 14759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:59.742475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33238 (local address 127.0.97.3:41913)
0620 23:38:59.752869 (+ 10394us) server_negotiation.cc:206] Beginning negotiation
0620 23:38:59.752879 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:38:59.754915 (+ 2036us) server_negotiation.cc:407] Connection header received
0620 23:38:59.755105 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:38:59.755113 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:38:59.755393 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:38:59.755605 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:38:59.756472 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:59.757656 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:59.760878 (+ 3222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:59.762222 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:59.762352 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:59.762580 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 23:38:59.763880 (+ 1300us) server_negotiation.cc:299] Negotiation successful
0620 23:38:59.764062 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10202,"thread_start_us":10034,"threads_started":1}
I20260620 23:38:59.763577 14754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:59.741415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45841 (local address 127.0.0.1:48372)
0620 23:38:59.754481 (+ 13066us) negotiation.cc:107] Waiting for socket to connect
0620 23:38:59.754500 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:38:59.754647 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:38:59.756192 (+ 1545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:38:59.756195 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:38:59.756218 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:38:59.756675 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:38:59.756690 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:38:59.758547 (+ 1857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:38:59.758555 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:38:59.760493 (+ 1938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:38:59.760511 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:38:59.762701 (+ 2190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:38:59.762706 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:38:59.762904 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:59.762969 (+ 65us) client_negotiation.cc:769] Sending connection context
0620 23:38:59.763045 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 23:38:59.763161 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12892,"mutex_wait_us":87,"thread_start_us":13064,"threads_started":1}
I20260620 23:38:59.765249 14758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:59.742162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48372 (local address 127.0.97.1:45841)
0620 23:38:59.755339 (+ 13177us) server_negotiation.cc:206] Beginning negotiation
0620 23:38:59.755347 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:38:59.755374 (+ 27us) server_negotiation.cc:407] Connection header received
0620 23:38:59.755486 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:38:59.755497 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:38:59.755710 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:38:59.755918 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:38:59.756890 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:59.758368 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:59.760669 (+ 2301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:59.762222 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:59.762360 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:59.762574 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0620 23:38:59.763718 (+ 1144us) server_negotiation.cc:299] Negotiation successful
0620 23:38:59.763889 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12643,"thread_start_us":12481,"threads_started":1}
I20260620 23:38:59.767992 14676 tablet_service.cc:1467] Processing CreateTablet for tablet d375e397abb84f99a1f160857c330dfd (DEFAULT_TABLE table=test-table [id=53a2ae787b384a8c9ed2c4458e6fe265]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:38:59.768085 14543 tablet_service.cc:1467] Processing CreateTablet for tablet d375e397abb84f99a1f160857c330dfd (DEFAULT_TABLE table=test-table [id=53a2ae787b384a8c9ed2c4458e6fe265]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:38:59.768781 14409 tablet_service.cc:1467] Processing CreateTablet for tablet d375e397abb84f99a1f160857c330dfd (DEFAULT_TABLE table=test-table [id=53a2ae787b384a8c9ed2c4458e6fe265]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:38:59.769348 14676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d375e397abb84f99a1f160857c330dfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:38:59.769456 14543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d375e397abb84f99a1f160857c330dfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:38:59.770026 14409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d375e397abb84f99a1f160857c330dfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:38:59.782269 14760 tablet_bootstrap.cc:492] T d375e397abb84f99a1f160857c330dfd P 8a1301fde2de4afe8548772edc8267a9: Bootstrap starting.
I20260620 23:38:59.784266 14761 tablet_bootstrap.cc:492] T d375e397abb84f99a1f160857c330dfd P a55a5d4f8a194f52b6da9351fdc8bdb1: Bootstrap starting.
I20260620 23:38:59.786588 14760 tablet_bootstrap.cc:654] T d375e397abb84f99a1f160857c330dfd P 8a1301fde2de4afe8548772edc8267a9: Neither blocks nor log segments found. Creating new log.
I20260620 23:38:59.787206 14762 tablet_bootstrap.cc:492] T d375e397abb84f99a1f160857c330dfd P acc9e1859c3e45aa9ab8428c27657281: Bootstrap starting.
I20260620 23:38:59.787771 14761 tablet_bootstrap.cc:654] T d375e397abb84f99a1f160857c330dfd P a55a5d4f8a194f52b6da9351fdc8bdb1: Neither blocks nor log segments found. Creating new log.
I20260620 23:38:59.788079 14760 log.cc:824] T d375e397abb84f99a1f160857c330dfd P 8a1301fde2de4afe8548772edc8267a9: Log is configured to *not* fsync() on all Append() calls
I20260620 23:38:59.789033 14761 log.cc:824] T d375e397abb84f99a1f160857c330dfd P a55a5d4f8a194f52b6da9351fdc8bdb1: Log is configured to *not* fsync() on all Append() calls
I20260620 23:38:59.790449 14762 tablet_bootstrap.cc:654] T d375e397abb84f99a1f160857c330dfd P acc9e1859c3e45aa9ab8428c27657281: Neither blocks nor log segments found. Creating new log.
I20260620 23:38:59.791667 14762 log.cc:824] T d375e397abb84f99a1f160857c330dfd P acc9e1859c3e45aa9ab8428c27657281: Log is configured to *not* fsync() on all Append() calls
I20260620 23:38:59.791918 14760 tablet_bootstrap.cc:492] T d375e397abb84f99a1f160857c330dfd P 8a1301fde2de4afe8548772edc8267a9: No bootstrap required, opened a new log
I20260620 23:38:59.792658 14760 ts_tablet_manager.cc:1397] T d375e397abb84f99a1f160857c330dfd P 8a1301fde2de4afe8548772edc8267a9: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260620 23:38:59.796737 14762 tablet_bootstrap.cc:492] T d375e397abb84f99a1f160857c330dfd P acc9e1859c3e45aa9ab8428c27657281: No bootstrap required, opened a new log
I20260620 23:38:59.796770 14761 tablet_bootstrap.cc:492] T d375e397abb84f99a1f160857c330dfd P a55a5d4f8a194f52b6da9351fdc8bdb1: No bootstrap required, opened a new log
I20260620 23:38:59.797010 14762 ts_tablet_manager.cc:1397] T d375e397abb84f99a1f160857c330dfd P acc9e1859c3e45aa9ab8428c27657281: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260620 23:38:59.797037 14761 ts_tablet_manager.cc:1397] T d375e397abb84f99a1f160857c330dfd P a55a5d4f8a194f52b6da9351fdc8bdb1: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.006s
I20260620 23:38:59.804611 14761 raft_consensus.cc:348] T d375e397abb84f99a1f160857c330dfd P a55a5d4f8a194f52b6da9351fdc8bdb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55a5d4f8a194f52b6da9351fdc8bdb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45841 } } peers { permanent_uuid: "acc9e1859c3e45aa9ab8428c27657281" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39353 } } peers { permanent_uuid: "8a1301fde2de4afe8548772edc8267a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41913 } }
I20260620 23:38:59.805061 14761 raft_consensus.cc:374] T d375e397abb84f99a1f160857c330dfd P a55a5d4f8a194f52b6da9351fdc8bdb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:38:59.805214 14761 raft_consensus.cc:729] T d375e397abb84f99a1f160857c330dfd P a55a5d4f8a194f52b6da9351fdc8bdb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a55a5d4f8a194f52b6da9351fdc8bdb1, State: Initialized, Role: FOLLOWER
I20260620 23:38:59.804991 14760 raft_consensus.cc:348] T d375e397abb84f99a1f160857c330dfd P 8a1301fde2de4afe8548772edc8267a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55a5d4f8a194f52b6da9351fdc8bdb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45841 } } peers { permanent_uuid: "acc9e1859c3e45aa9ab8428c27657281" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39353 } } peers { permanent_uuid: "8a1301fde2de4afe8548772edc8267a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41913 } }
I20260620 23:38:59.805501 14760 raft_consensus.cc:374] T d375e397abb84f99a1f160857c330dfd P 8a1301fde2de4afe8548772edc8267a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:38:59.805625 14760 raft_consensus.cc:729] T d375e397abb84f99a1f160857c330dfd P 8a1301fde2de4afe8548772edc8267a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a1301fde2de4afe8548772edc8267a9, State: Initialized, Role: FOLLOWER
I20260620 23:38:59.805974 14761 consensus_queue.cc:260] T d375e397abb84f99a1f160857c330dfd P a55a5d4f8a194f52b6da9351fdc8bdb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55a5d4f8a194f52b6da9351fdc8bdb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45841 } } peers { permanent_uuid: "acc9e1859c3e45aa9ab8428c27657281" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39353 } } peers { permanent_uuid: "8a1301fde2de4afe8548772edc8267a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41913 } }
I20260620 23:38:59.806247 14760 consensus_queue.cc:260] T d375e397abb84f99a1f160857c330dfd P 8a1301fde2de4afe8548772edc8267a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55a5d4f8a194f52b6da9351fdc8bdb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45841 } } peers { permanent_uuid: "acc9e1859c3e45aa9ab8428c27657281" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39353 } } peers { permanent_uuid: "8a1301fde2de4afe8548772edc8267a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41913 } }
I20260620 23:38:59.808058 14762 raft_consensus.cc:348] T d375e397abb84f99a1f160857c330dfd P acc9e1859c3e45aa9ab8428c27657281 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55a5d4f8a194f52b6da9351fdc8bdb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45841 } } peers { permanent_uuid: "acc9e1859c3e45aa9ab8428c27657281" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39353 } } peers { permanent_uuid: "8a1301fde2de4afe8548772edc8267a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41913 } }
I20260620 23:38:59.808717 14762 raft_consensus.cc:374] T d375e397abb84f99a1f160857c330dfd P acc9e1859c3e45aa9ab8428c27657281 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:38:59.808866 14762 raft_consensus.cc:729] T d375e397abb84f99a1f160857c330dfd P acc9e1859c3e45aa9ab8428c27657281 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc9e1859c3e45aa9ab8428c27657281, State: Initialized, Role: FOLLOWER
I20260620 23:38:59.809716 14762 consensus_queue.cc:260] T d375e397abb84f99a1f160857c330dfd P acc9e1859c3e45aa9ab8428c27657281 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55a5d4f8a194f52b6da9351fdc8bdb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45841 } } peers { permanent_uuid: "acc9e1859c3e45aa9ab8428c27657281" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39353 } } peers { permanent_uuid: "8a1301fde2de4afe8548772edc8267a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41913 } }
I20260620 23:38:59.810015 14761 ts_tablet_manager.cc:1428] T d375e397abb84f99a1f160857c330dfd P a55a5d4f8a194f52b6da9351fdc8bdb1: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260620 23:38:59.811095 14608 heartbeater.cc:503] Master 127.0.97.62:41617 was elected leader, sending a full tablet report...
I20260620 23:38:59.812059 14762 ts_tablet_manager.cc:1428] T d375e397abb84f99a1f160857c330dfd P acc9e1859c3e45aa9ab8428c27657281: Time spent starting tablet: real 0.015s user 0.015s sys 0.001s
I20260620 23:38:59.813122 14741 heartbeater.cc:503] Master 127.0.97.62:41617 was elected leader, sending a full tablet report...
I20260620 23:38:59.814704 14760 ts_tablet_manager.cc:1428] T d375e397abb84f99a1f160857c330dfd P 8a1301fde2de4afe8548772edc8267a9: Time spent starting tablet: real 0.022s user 0.015s sys 0.004s
W20260620 23:38:59.869235 14475 tablet.cc:2369] T d375e397abb84f99a1f160857c330dfd P a55a5d4f8a194f52b6da9351fdc8bdb1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:38:59.887498 14609 tablet.cc:2369] T d375e397abb84f99a1f160857c330dfd P acc9e1859c3e45aa9ab8428c27657281: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:38:59.916610 14742 tablet.cc:2369] T d375e397abb84f99a1f160857c330dfd P 8a1301fde2de4afe8548772edc8267a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:38:59.928833 14767 raft_consensus.cc:482] T d375e397abb84f99a1f160857c330dfd P 8a1301fde2de4afe8548772edc8267a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:38:59.929171 14767 raft_consensus.cc:504] T d375e397abb84f99a1f160857c330dfd P 8a1301fde2de4afe8548772edc8267a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55a5d4f8a194f52b6da9351fdc8bdb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45841 } } peers { permanent_uuid: "acc9e1859c3e45aa9ab8428c27657281" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39353 } } peers { permanent_uuid: "8a1301fde2de4afe8548772edc8267a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41913 } }
I20260620 23:38:59.931124 14767 leader_election.cc:288] T d375e397abb84f99a1f160857c330dfd P 8a1301fde2de4afe8548772edc8267a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a55a5d4f8a194f52b6da9351fdc8bdb1 (127.0.97.1:45841), acc9e1859c3e45aa9ab8428c27657281 (127.0.97.2:39353)
I20260620 23:38:59.937891 14758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:59.931481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54119 (local address 127.0.97.1:45841)
0620 23:38:59.931696 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 23:38:59.931701 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:38:59.931726 (+ 25us) server_negotiation.cc:407] Connection header received
0620 23:38:59.931872 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:38:59.931882 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:38:59.931977 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:38:59.932087 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:38:59.932608 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:59.933556 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:59.935945 (+ 2389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:59.936966 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:59.937070 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:59.937146 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 23:38:59.937507 (+ 361us) server_negotiation.cc:299] Negotiation successful
0620 23:38:59.937600 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 23:38:59.937925 14744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:59.931311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45841 (local address 127.0.97.3:54119)
0620 23:38:59.931556 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 23:38:59.931578 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 23:38:59.931721 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:38:59.932200 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:38:59.932204 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:38:59.932232 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:38:59.932457 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:38:59.932468 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:38:59.933706 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:38:59.933709 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:38:59.935756 (+ 2047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:38:59.935770 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:38:59.937137 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:38:59.937144 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:38:59.937335 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:59.937372 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 23:38:59.937477 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 23:38:59.937610 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73,"mutex_wait_us":68}
I20260620 23:38:59.938608 14757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:59.931866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56675 (local address 127.0.97.2:39353)
0620 23:38:59.932091 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 23:38:59.932098 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:38:59.932498 (+ 400us) server_negotiation.cc:407] Connection header received
0620 23:38:59.932653 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:38:59.932660 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:38:59.932741 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:38:59.932838 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:38:59.933327 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:59.934130 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:59.936558 (+ 2428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:59.937578 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:59.937704 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:59.937802 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 23:38:59.938216 (+ 414us) server_negotiation.cc:299] Negotiation successful
0620 23:38:59.938361 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 23:38:59.938661 14769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:59.931585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39353 (local address 127.0.97.3:56675)
0620 23:38:59.932366 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0620 23:38:59.932395 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 23:38:59.932517 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:38:59.932891 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:38:59.932894 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:38:59.932913 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:38:59.933172 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:38:59.933178 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:38:59.934292 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:38:59.934298 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:38:59.936395 (+ 2097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:38:59.936408 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:38:59.937795 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:38:59.937801 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:38:59.937995 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:59.938032 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 23:38:59.938174 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 23:38:59.938307 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"mutex_wait_us":26,"thread_start_us":412,"threads_started":1}
I20260620 23:38:59.939093 14429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d375e397abb84f99a1f160857c330dfd" candidate_uuid: "8a1301fde2de4afe8548772edc8267a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a55a5d4f8a194f52b6da9351fdc8bdb1" is_pre_election: true
I20260620 23:38:59.939637 14429 raft_consensus.cc:2454] T d375e397abb84f99a1f160857c330dfd P a55a5d4f8a194f52b6da9351fdc8bdb1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a1301fde2de4afe8548772edc8267a9 in term 0.
I20260620 23:38:59.939925 14563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d375e397abb84f99a1f160857c330dfd" candidate_uuid: "8a1301fde2de4afe8548772edc8267a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc9e1859c3e45aa9ab8428c27657281" is_pre_election: true
I20260620 23:38:59.940431 14632 leader_election.cc:302] T d375e397abb84f99a1f160857c330dfd P 8a1301fde2de4afe8548772edc8267a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a1301fde2de4afe8548772edc8267a9, a55a5d4f8a194f52b6da9351fdc8bdb1; no voters:
I20260620 23:38:59.940500 14563 raft_consensus.cc:2454] T d375e397abb84f99a1f160857c330dfd P acc9e1859c3e45aa9ab8428c27657281 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a1301fde2de4afe8548772edc8267a9 in term 0.
I20260620 23:38:59.940889 14767 raft_consensus.cc:2781] T d375e397abb84f99a1f160857c330dfd P 8a1301fde2de4afe8548772edc8267a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:38:59.941058 14767 raft_consensus.cc:482] T d375e397abb84f99a1f160857c330dfd P 8a1301fde2de4afe8548772edc8267a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:38:59.941177 14767 raft_consensus.cc:3037] T d375e397abb84f99a1f160857c330dfd P 8a1301fde2de4afe8548772edc8267a9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:38:59.943535 14767 raft_consensus.cc:504] T d375e397abb84f99a1f160857c330dfd P 8a1301fde2de4afe8548772edc8267a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55a5d4f8a194f52b6da9351fdc8bdb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45841 } } peers { permanent_uuid: "acc9e1859c3e45aa9ab8428c27657281" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39353 } } peers { permanent_uuid: "8a1301fde2de4afe8548772edc8267a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41913 } }
I20260620 23:38:59.944121 14767 leader_election.cc:288] T d375e397abb84f99a1f160857c330dfd P 8a1301fde2de4afe8548772edc8267a9 [CANDIDATE]: Term 1 election: Requested vote from peers a55a5d4f8a194f52b6da9351fdc8bdb1 (127.0.97.1:45841), acc9e1859c3e45aa9ab8428c27657281 (127.0.97.2:39353)
I20260620 23:38:59.944656 14563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d375e397abb84f99a1f160857c330dfd" candidate_uuid: "8a1301fde2de4afe8548772edc8267a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc9e1859c3e45aa9ab8428c27657281"
I20260620 23:38:59.944693 14429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d375e397abb84f99a1f160857c330dfd" candidate_uuid: "8a1301fde2de4afe8548772edc8267a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a55a5d4f8a194f52b6da9351fdc8bdb1"
I20260620 23:38:59.944861 14563 raft_consensus.cc:3037] T d375e397abb84f99a1f160857c330dfd P acc9e1859c3e45aa9ab8428c27657281 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:38:59.944952 14429 raft_consensus.cc:3037] T d375e397abb84f99a1f160857c330dfd P a55a5d4f8a194f52b6da9351fdc8bdb1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:38:59.947278 14563 raft_consensus.cc:2454] T d375e397abb84f99a1f160857c330dfd P acc9e1859c3e45aa9ab8428c27657281 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a1301fde2de4afe8548772edc8267a9 in term 1.
I20260620 23:38:59.947288 14429 raft_consensus.cc:2454] T d375e397abb84f99a1f160857c330dfd P a55a5d4f8a194f52b6da9351fdc8bdb1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a1301fde2de4afe8548772edc8267a9 in term 1.
I20260620 23:38:59.947754 14629 leader_election.cc:302] T d375e397abb84f99a1f160857c330dfd P 8a1301fde2de4afe8548772edc8267a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a1301fde2de4afe8548772edc8267a9, acc9e1859c3e45aa9ab8428c27657281; no voters:
I20260620 23:38:59.948122 14767 raft_consensus.cc:2781] T d375e397abb84f99a1f160857c330dfd P 8a1301fde2de4afe8548772edc8267a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:38:59.948608 14767 raft_consensus.cc:686] T d375e397abb84f99a1f160857c330dfd P 8a1301fde2de4afe8548772edc8267a9 [term 1 LEADER]: Becoming Leader. State: Replica: 8a1301fde2de4afe8548772edc8267a9, State: Running, Role: LEADER
I20260620 23:38:59.949110 14767 consensus_queue.cc:237] T d375e397abb84f99a1f160857c330dfd P 8a1301fde2de4afe8548772edc8267a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55a5d4f8a194f52b6da9351fdc8bdb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45841 } } peers { permanent_uuid: "acc9e1859c3e45aa9ab8428c27657281" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39353 } } peers { permanent_uuid: "8a1301fde2de4afe8548772edc8267a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41913 } }
I20260620 23:38:59.953819 14283 catalog_manager.cc:5495] T d375e397abb84f99a1f160857c330dfd P 8a1301fde2de4afe8548772edc8267a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a1301fde2de4afe8548772edc8267a9 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "8a1301fde2de4afe8548772edc8267a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55a5d4f8a194f52b6da9351fdc8bdb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acc9e1859c3e45aa9ab8428c27657281" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a1301fde2de4afe8548772edc8267a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41913 } health_report { overall_health: HEALTHY } } }
I20260620 23:38:59.994594 14759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:38:59.988890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33252 (local address 127.0.97.3:41913)
0620 23:38:59.989135 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 23:38:59.989141 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:38:59.989250 (+ 109us) server_negotiation.cc:407] Connection header received
0620 23:38:59.989520 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:38:59.989524 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:38:59.989604 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:38:59.989764 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:38:59.990165 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:59.990908 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:59.992389 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:38:59.992810 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:38:59.992948 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:38:59.993480 (+ 532us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:38:59.994003 (+ 523us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:38:59.994178 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 23:38:59.994231 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 23:38:59.994312 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260620 23:39:00.004133 14563 raft_consensus.cc:1261] T d375e397abb84f99a1f160857c330dfd P acc9e1859c3e45aa9ab8428c27657281 [term 1 FOLLOWER]: Refusing update from remote peer 8a1301fde2de4afe8548772edc8267a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:39:00.004133 14429 raft_consensus.cc:1261] T d375e397abb84f99a1f160857c330dfd P a55a5d4f8a194f52b6da9351fdc8bdb1 [term 1 FOLLOWER]: Refusing update from remote peer 8a1301fde2de4afe8548772edc8267a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:39:00.005344 14770 consensus_queue.cc:1035] T d375e397abb84f99a1f160857c330dfd P 8a1301fde2de4afe8548772edc8267a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a55a5d4f8a194f52b6da9351fdc8bdb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:39:00.005746 14767 consensus_queue.cc:1035] T d375e397abb84f99a1f160857c330dfd P 8a1301fde2de4afe8548772edc8267a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc9e1859c3e45aa9ab8428c27657281" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:39:00.018208 14773 mvcc.cc:205] Tried to move back new op lower bound from 7299066839045234688 to 7299066838835458048. Current Snapshot: MvccSnapshot[applied={T|T < 7299066839045234688}]
I20260620 23:39:00.019371 14775 mvcc.cc:205] Tried to move back new op lower bound from 7299066839045234688 to 7299066838835458048. Current Snapshot: MvccSnapshot[applied={T|T < 7299066839045234688}]
I20260620 23:39:00.021219 14774 mvcc.cc:205] Tried to move back new op lower bound from 7299066839045234688 to 7299066838835458048. Current Snapshot: MvccSnapshot[applied={T|T < 7299066839045234688}]
I20260620 23:39:00.035221 14285 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:39:00.035530 14285 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:39:00.039218 14285 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f7e7ba8cc23b4fdbaf0a02f1a75e37b9: Sending DeleteTablet for 3 replicas of tablet d375e397abb84f99a1f160857c330dfd
I20260620 23:39:00.040455 14543 tablet_service.cc:1514] Processing DeleteTablet for tablet d375e397abb84f99a1f160857c330dfd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:39:00 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46004
I20260620 23:39:00.040457 14676 tablet_service.cc:1514] Processing DeleteTablet for tablet d375e397abb84f99a1f160857c330dfd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:39:00 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33238
I20260620 23:39:00.040493 14409 tablet_service.cc:1514] Processing DeleteTablet for tablet d375e397abb84f99a1f160857c330dfd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:39:00 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48372
I20260620 23:39:00.042367 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 14346
I20260620 23:39:00.053129 14782 tablet_replica.cc:330] T d375e397abb84f99a1f160857c330dfd P acc9e1859c3e45aa9ab8428c27657281: stopping tablet replica
I20260620 23:39:00.053673 14782 raft_consensus.cc:2229] T d375e397abb84f99a1f160857c330dfd P acc9e1859c3e45aa9ab8428c27657281 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:39:00.054245 14782 raft_consensus.cc:2258] T d375e397abb84f99a1f160857c330dfd P acc9e1859c3e45aa9ab8428c27657281 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:39:00.055166 14783 tablet_replica.cc:330] T d375e397abb84f99a1f160857c330dfd P 8a1301fde2de4afe8548772edc8267a9: stopping tablet replica
I20260620 23:39:00.055567 14783 raft_consensus.cc:2229] T d375e397abb84f99a1f160857c330dfd P 8a1301fde2de4afe8548772edc8267a9 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:39:00.056345 14783 raft_consensus.cc:2258] T d375e397abb84f99a1f160857c330dfd P 8a1301fde2de4afe8548772edc8267a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:39:00.059023 14782 ts_tablet_manager.cc:1905] T d375e397abb84f99a1f160857c330dfd P acc9e1859c3e45aa9ab8428c27657281: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:39:00.059217 14783 ts_tablet_manager.cc:1905] T d375e397abb84f99a1f160857c330dfd P 8a1301fde2de4afe8548772edc8267a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:39:00.063907 14784 tablet_replica.cc:330] T d375e397abb84f99a1f160857c330dfd P a55a5d4f8a194f52b6da9351fdc8bdb1: stopping tablet replica
I20260620 23:39:00.064579 14784 raft_consensus.cc:2229] T d375e397abb84f99a1f160857c330dfd P a55a5d4f8a194f52b6da9351fdc8bdb1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:39:00.065220 14784 raft_consensus.cc:2258] T d375e397abb84f99a1f160857c330dfd P a55a5d4f8a194f52b6da9351fdc8bdb1 [term 1 FOLLOWER]: Raft consensus is shut down!
Jun 20 23:39:00 dist-test-slave-tcgx krb5kdc[14240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 23:39:00.066960 14782 ts_tablet_manager.cc:1918] T d375e397abb84f99a1f160857c330dfd P acc9e1859c3e45aa9ab8428c27657281: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:39:00.069168 14784 ts_tablet_manager.cc:1905] T d375e397abb84f99a1f160857c330dfd P a55a5d4f8a194f52b6da9351fdc8bdb1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:39:00.069433 14782 log.cc:1196] T d375e397abb84f99a1f160857c330dfd P acc9e1859c3e45aa9ab8428c27657281: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-1/wal/wals/d375e397abb84f99a1f160857c330dfd
I20260620 23:39:00.070080 14782 ts_tablet_manager.cc:1939] T d375e397abb84f99a1f160857c330dfd P acc9e1859c3e45aa9ab8428c27657281: Deleting consensus metadata
I20260620 23:39:00.074069 14783 ts_tablet_manager.cc:1918] T d375e397abb84f99a1f160857c330dfd P 8a1301fde2de4afe8548772edc8267a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:39:00.074460 14783 log.cc:1196] T d375e397abb84f99a1f160857c330dfd P 8a1301fde2de4afe8548772edc8267a9: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-2/wal/wals/d375e397abb84f99a1f160857c330dfd
I20260620 23:39:00.075189 14783 ts_tablet_manager.cc:1939] T d375e397abb84f99a1f160857c330dfd P 8a1301fde2de4afe8548772edc8267a9: Deleting consensus metadata
I20260620 23:39:00.076253 14758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:00.043348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48382 (local address 127.0.97.1:45841)
0620 23:39:00.043613 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0620 23:39:00.043622 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:39:00.055651 (+ 12029us) server_negotiation.cc:407] Connection header received
0620 23:39:00.055917 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:39:00.055922 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:39:00.056054 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:39:00.056178 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:39:00.058370 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:00.059552 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:00.061549 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:00.062215 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:00.062312 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:00.068302 (+ 5990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:39:00.068315 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:39:00.068332 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:39:00.068379 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:39:00.072340 (+ 3961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:39:00.073312 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:39:00.073321 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:39:00.073325 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:39:00.073505 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:39:00.074001 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:39:00.074005 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:39:00.074008 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:39:00.074588 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:39:00.074896 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0620 23:39:00.075555 (+ 659us) server_negotiation.cc:299] Negotiation successful
0620 23:39:00.075832 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 23:39:00.077472 14784 ts_tablet_manager.cc:1918] T d375e397abb84f99a1f160857c330dfd P a55a5d4f8a194f52b6da9351fdc8bdb1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:39:00.078034 14271 catalog_manager.cc:4841] TS 8a1301fde2de4afe8548772edc8267a9 (127.0.97.3:41913): tablet d375e397abb84f99a1f160857c330dfd (table test-table [id=53a2ae787b384a8c9ed2c4458e6fe265]) successfully deleted
I20260620 23:39:00.077879 14469 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:39:00.079659 14269 catalog_manager.cc:4841] TS acc9e1859c3e45aa9ab8428c27657281 (127.0.97.2:39353): tablet d375e397abb84f99a1f160857c330dfd (table test-table [id=53a2ae787b384a8c9ed2c4458e6fe265]) successfully deleted
I20260620 23:39:00.084411 14784 log.cc:1196] T d375e397abb84f99a1f160857c330dfd P a55a5d4f8a194f52b6da9351fdc8bdb1: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-0/wal/wals/d375e397abb84f99a1f160857c330dfd
I20260620 23:39:00.172081 14784 ts_tablet_manager.cc:1939] T d375e397abb84f99a1f160857c330dfd P a55a5d4f8a194f52b6da9351fdc8bdb1: Deleting consensus metadata
I20260620 23:39:00.172299 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 14346
W20260620 23:39:00.198426 14272 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:45841 (error 108)
I20260620 23:39:00.199182 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 14480
I20260620 23:39:00.200695 14754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:00.200053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45841 (local address 127.0.0.1:48398)
0620 23:39:00.200376 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:00.200508 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216}
W20260620 23:39:00.200963 14272 catalog_manager.cc:4584] TS a55a5d4f8a194f52b6da9351fdc8bdb1 (127.0.97.1:45841): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d375e397abb84f99a1f160857c330dfd: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45841: connect: Connection refused (error 111)
Jun 20 23:39:00 dist-test-slave-tcgx krb5kdc[14240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 23:39:00.218407 14757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:00.200159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46010 (local address 127.0.97.2:39353)
0620 23:39:00.200366 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 23:39:00.200374 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:39:00.200509 (+ 135us) server_negotiation.cc:407] Connection header received
0620 23:39:00.200686 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:39:00.200693 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:39:00.200796 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:39:00.200926 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:39:00.202332 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:00.203648 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:00.205075 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:00.205770 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:00.205912 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:00.210742 (+ 4830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:39:00.210766 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:39:00.210787 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:39:00.210840 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:39:00.215152 (+ 4312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:39:00.215933 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:39:00.215944 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:39:00.215948 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:39:00.216117 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:39:00.216503 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:39:00.216510 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:39:00.216516 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:39:00.217082 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:39:00.217427 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0620 23:39:00.217718 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 23:39:00.217971 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 23:39:00.219247 14603 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:39:00.254776 14754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:00.254292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45841 (local address 127.0.0.1:48404)
0620 23:39:00.254522 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:00.254615 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260620 23:39:00.255936 14754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:00.255569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45841 (local address 127.0.0.1:48418)
0620 23:39:00.255733 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:00.255815 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 23:39:00.316414 14754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:00.315928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45841 (local address 127.0.0.1:48430)
0620 23:39:00.316167 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:00.316269 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260620 23:39:00.317569 14754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:00.317208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45841 (local address 127.0.0.1:48436)
0620 23:39:00.317363 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:00.317442 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 23:39:00.362828 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 14480
I20260620 23:39:00.381105 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 14613
Jun 20 23:39:00 dist-test-slave-tcgx krb5kdc[14240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 23:39:00.398592 14754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:00.397928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45841 (local address 127.0.0.1:48444)
0620 23:39:00.398242 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:00.398370 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182}
I20260620 23:39:00.398749 14759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:00.382059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33266 (local address 127.0.97.3:41913)
0620 23:39:00.382275 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 23:39:00.382280 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:39:00.382301 (+ 21us) server_negotiation.cc:407] Connection header received
0620 23:39:00.382525 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:39:00.382531 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:39:00.382620 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:39:00.382740 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:39:00.384133 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:00.384980 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:00.386355 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:00.386833 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:00.387109 (+ 276us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:00.392082 (+ 4973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:39:00.392095 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:39:00.392109 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:39:00.392152 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:39:00.395599 (+ 3447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:39:00.396345 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:39:00.396361 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:39:00.396365 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:39:00.396551 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:39:00.396981 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:39:00.396988 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:39:00.396993 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:39:00.397356 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:39:00.397705 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0620 23:39:00.398028 (+ 323us) server_negotiation.cc:299] Negotiation successful
0620 23:39:00.398257 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 23:39:00.399735 14736 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:39:00.400004 14754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:00.399498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45841 (local address 127.0.0.1:48452)
0620 23:39:00.399705 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:00.399786 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260620 23:39:00.522094 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 14613
I20260620 23:39:00.543838 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 14253
I20260620 23:39:00.545125 14315 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:39:00.659654 14754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:00.659018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45841 (local address 127.0.0.1:48460)
0620 23:39:00.659264 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:00.659384 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260620 23:39:00.659958 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 14253
I20260620 23:39:00.682004 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41617
--webserver_interface=127.0.97.62
--webserver_port=44471
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41617
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:39:01.026175 14791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:39:01.026489 14791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:39:01.026687 14791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:39:01.034284 14791 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:39:01.034406 14791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:39:01.034523 14791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:39:01.034634 14791 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:39:01.034720 14791 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:39:01.045154 14791 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41617
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=44471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:39:01.046571 14791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:39:01.047976 14791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:39:01.071676 14798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:39:01.071702 14800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:39:01.071738 14797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:39:01.072443 14791 server_base.cc:1034] running on GCE node
I20260620 23:39:01.073417 14791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:39:01.075614 14791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:39:01.076936 14791 hybrid_clock.cc:648] HybridClock initialized: now 1781998741076898 us; error 60 us; skew 500 ppm
I20260620 23:39:01.077391 14791 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:39:01 dist-test-slave-tcgx krb5kdc[14240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998741, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:39:01.081760 14791 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 23:39:01.083711 14791 webserver.cc:460] Webserver started at http://127.0.97.62:44471/ using document root <none> and password file <none>
I20260620 23:39:01.084479 14791 fs_manager.cc:359] Metadata directory not provided
I20260620 23:39:01.084604 14791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:39:01.090910 14791 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 23:39:01.094077 14807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:39:01.095678 14791 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 23:39:01.095896 14791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/master-0/wal
uuid: "f7e7ba8cc23b4fdbaf0a02f1a75e37b9"
format_stamp: "Formatted at 2026-06-20 23:38:57 on dist-test-slave-tcgx"
server_key: "b3b53e6a0c350ff5e38b66f00ea6834e"
server_key_iv: "2503cbea121e0a69915fc15e3a7ce45b"
server_key_version: "encryptionkey@0"
I20260620 23:39:01.097110 14791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:39:01.123157 14791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:39:01.128196 14791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:39:01.128450 14791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:39:01.148545 14791 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41617
I20260620 23:39:01.148568 14858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41617 every 8 connection(s)
I20260620 23:39:01.150184 14791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:39:01.157718 14859 sys_catalog.cc:263] Verifying existing consensus state
I20260620 23:39:01.157943 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 14791
I20260620 23:39:01.158973 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:45841
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:41617
--webserver_port=34901
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:39:01.163317 14859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7e7ba8cc23b4fdbaf0a02f1a75e37b9: Bootstrap starting.
I20260620 23:39:01.185496 14859 log.cc:824] T 00000000000000000000000000000000 P f7e7ba8cc23b4fdbaf0a02f1a75e37b9: Log is configured to *not* fsync() on all Append() calls
I20260620 23:39:01.198575 14859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7e7ba8cc23b4fdbaf0a02f1a75e37b9: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:39:01.199256 14859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7e7ba8cc23b4fdbaf0a02f1a75e37b9: Bootstrap complete.
I20260620 23:39:01.209857 14859 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7e7ba8cc23b4fdbaf0a02f1a75e37b9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7e7ba8cc23b4fdbaf0a02f1a75e37b9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41617 } }
I20260620 23:39:01.210793 14859 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7e7ba8cc23b4fdbaf0a02f1a75e37b9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7e7ba8cc23b4fdbaf0a02f1a75e37b9, State: Initialized, Role: FOLLOWER
I20260620 23:39:01.211519 14859 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7e7ba8cc23b4fdbaf0a02f1a75e37b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7e7ba8cc23b4fdbaf0a02f1a75e37b9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41617 } }
I20260620 23:39:01.211782 14859 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7e7ba8cc23b4fdbaf0a02f1a75e37b9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:39:01.211920 14859 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7e7ba8cc23b4fdbaf0a02f1a75e37b9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:39:01.212123 14859 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7e7ba8cc23b4fdbaf0a02f1a75e37b9 [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:39:01.216250 14859 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7e7ba8cc23b4fdbaf0a02f1a75e37b9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7e7ba8cc23b4fdbaf0a02f1a75e37b9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41617 } }
I20260620 23:39:01.216818 14859 leader_election.cc:302] T 00000000000000000000000000000000 P f7e7ba8cc23b4fdbaf0a02f1a75e37b9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7e7ba8cc23b4fdbaf0a02f1a75e37b9; no voters:
I20260620 23:39:01.217479 14859 leader_election.cc:288] T 00000000000000000000000000000000 P f7e7ba8cc23b4fdbaf0a02f1a75e37b9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 23:39:01.217619 14863 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7e7ba8cc23b4fdbaf0a02f1a75e37b9 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 23:39:01.218926 14863 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7e7ba8cc23b4fdbaf0a02f1a75e37b9 [term 2 LEADER]: Becoming Leader. State: Replica: f7e7ba8cc23b4fdbaf0a02f1a75e37b9, State: Running, Role: LEADER
I20260620 23:39:01.219615 14863 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7e7ba8cc23b4fdbaf0a02f1a75e37b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7e7ba8cc23b4fdbaf0a02f1a75e37b9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41617 } }
I20260620 23:39:01.220088 14859 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7e7ba8cc23b4fdbaf0a02f1a75e37b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:39:01.222152 14865 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7e7ba8cc23b4fdbaf0a02f1a75e37b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7e7ba8cc23b4fdbaf0a02f1a75e37b9. Latest consensus state: current_term: 2 leader_uuid: "f7e7ba8cc23b4fdbaf0a02f1a75e37b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7e7ba8cc23b4fdbaf0a02f1a75e37b9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41617 } } }
I20260620 23:39:01.222879 14865 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7e7ba8cc23b4fdbaf0a02f1a75e37b9 [sys.catalog]: This master's current role is: LEADER
I20260620 23:39:01.222563 14864 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7e7ba8cc23b4fdbaf0a02f1a75e37b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f7e7ba8cc23b4fdbaf0a02f1a75e37b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7e7ba8cc23b4fdbaf0a02f1a75e37b9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41617 } } }
I20260620 23:39:01.223412 14864 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7e7ba8cc23b4fdbaf0a02f1a75e37b9 [sys.catalog]: This master's current role is: LEADER
I20260620 23:39:01.228530 14869 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:39:01.238579 14869 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:39:01.240680 14869 catalog_manager.cc:1259] Loaded cluster ID: 7523bac8126c481e9cfe3986e84b2d5f
I20260620 23:39:01.240864 14869 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:39:01.245146 14869 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 23:39:01.271893 14869 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 23:39:01.283305 14869 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:39:01.284740 14869 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f7e7ba8cc23b4fdbaf0a02f1a75e37b9: Loaded TSK: 0
I20260620 23:39:01.285919 14869 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 23:39:01.557179 14861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:39:01.557533 14861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:39:01.557698 14861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:39:01.565447 14861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:39:01.565624 14861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:39:01.565797 14861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 23:39:01.576211 14861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:45841
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=34901
--tserver_master_addrs=127.0.97.62:41617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:39:01.577605 14861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:39:01.579033 14861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:39:01.594861 14887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:39:01.595366 14886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:39:01.595553 14861 server_base.cc:1034] running on GCE node
W20260620 23:39:01.594774 14889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:39:01.596453 14861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:39:01.597568 14861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:39:01.598801 14861 hybrid_clock.cc:648] HybridClock initialized: now 1781998741598756 us; error 41 us; skew 500 ppm
I20260620 23:39:01.599174 14861 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:39:01 dist-test-slave-tcgx krb5kdc[14240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998741, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:39:01.606359 14861 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 23:39:01.609987 14861 webserver.cc:460] Webserver started at http://127.0.97.1:34901/ using document root <none> and password file <none>
I20260620 23:39:01.611028 14861 fs_manager.cc:359] Metadata directory not provided
I20260620 23:39:01.611202 14861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:39:01.618172 14861 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 23:39:01.621413 14896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:39:01.623020 14861 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:39:01.623208 14861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-0/wal
uuid: "a55a5d4f8a194f52b6da9351fdc8bdb1"
format_stamp: "Formatted at 2026-06-20 23:38:58 on dist-test-slave-tcgx"
server_key: "1c58a1c8ed30db672d5e9ccb3c142f8e"
server_key_iv: "b1cbda2f23d38799a6c9e3eacaa16b46"
server_key_version: "encryptionkey@0"
I20260620 23:39:01.624235 14861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:39:01.644023 14861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:39:01.648955 14861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:39:01.649257 14861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:39:01.650324 14861 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:39:01.651912 14861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:39:01.652053 14861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:39:01.652204 14861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:39:01.652300 14861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:39:01.687597 14861 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45841
I20260620 23:39:01.687631 15008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45841 every 8 connection(s)
I20260620 23:39:01.689108 14861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
I20260620 23:39:01.696988 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 14861
I20260620 23:39:01.697669 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:39353
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:41617
--webserver_port=40057
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:39:01 dist-test-slave-tcgx krb5kdc[14240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998741, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:39:01.722362 15011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:01.692198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48453 (local address 127.0.97.62:41617)
0620 23:39:01.692855 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0620 23:39:01.692867 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:39:01.693599 (+ 732us) server_negotiation.cc:407] Connection header received
0620 23:39:01.695916 (+ 2317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:39:01.695930 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:39:01.696347 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:39:01.696797 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:39:01.698587 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:01.705496 (+ 6909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:01.708523 (+ 3027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:01.709243 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:01.709394 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:01.714269 (+ 4875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:39:01.714279 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:39:01.714297 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:39:01.714344 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:39:01.717280 (+ 2936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:39:01.718368 (+ 1088us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:39:01.718374 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:39:01.718380 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:39:01.718543 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:39:01.719124 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:39:01.719128 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:39:01.719130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:39:01.719607 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:39:01.719762 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 23:39:01.720816 (+ 1054us) server_negotiation.cc:299] Negotiation successful
0620 23:39:01.721319 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":349,"threads_started":1}
I20260620 23:39:01.722782 15012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:01.692661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41617 (local address 127.0.97.1:48453)
0620 23:39:01.693406 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:01.693447 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 23:39:01.695285 (+ 1838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:39:01.696998 (+ 1713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:39:01.697010 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:39:01.697609 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:39:01.698308 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:39:01.698327 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:39:01.705767 (+ 7440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:39:01.705775 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:39:01.708281 (+ 2506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:39:01.708307 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:39:01.709537 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:39:01.709548 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:39:01.709888 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:01.710707 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:39:01.710747 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:39:01.713933 (+ 3186us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:39:01.717459 (+ 3526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:39:01.717472 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:39:01.717494 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:39:01.718215 (+ 721us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:39:01.718678 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:39:01.718686 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:39:01.718691 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:39:01.719000 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:39:01.719755 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:39:01.719766 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:39:01.720285 (+ 519us) client_negotiation.cc:769] Sending connection context
0620 23:39:01.720633 (+ 348us) client_negotiation.cc:240] Negotiation successful
0620 23:39:01.721307 (+ 674us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":296,"threads_started":1}
I20260620 23:39:01.726104 15009 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41617
I20260620 23:39:01.726435 15009 heartbeater.cc:464] Registering TS with master...
I20260620 23:39:01.727370 15009 heartbeater.cc:511] Master 127.0.97.62:41617 requested a full tablet report, sending...
I20260620 23:39:01.729822 14824 ts_manager.cc:194] Registered new tserver with Master: a55a5d4f8a194f52b6da9351fdc8bdb1 (127.0.97.1:45841)
I20260620 23:39:01.731354 14824 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:48453
W20260620 23:39:02.040848 15013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:39:02.041162 15013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:39:02.041258 15013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:39:02.048626 15013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:39:02.048748 15013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:39:02.048866 15013 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 23:39:02.058679 15013 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:39353
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=40057
--tserver_master_addrs=127.0.97.62:41617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:39:02.060083 15013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:39:02.061378 15013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:39:02.075259 15018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:39:02.075295 15019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:39:02.075523 15021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:39:02.075917 15013 server_base.cc:1034] running on GCE node
I20260620 23:39:02.076540 15013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:39:02.077528 15013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:39:02.078768 15013 hybrid_clock.cc:648] HybridClock initialized: now 1781998742078721 us; error 60 us; skew 500 ppm
I20260620 23:39:02.079157 15013 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:39:02 dist-test-slave-tcgx krb5kdc[14240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998742, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:39:02.082273 15013 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 23:39:02.083781 15013 webserver.cc:460] Webserver started at http://127.0.97.2:40057/ using document root <none> and password file <none>
I20260620 23:39:02.084493 15013 fs_manager.cc:359] Metadata directory not provided
I20260620 23:39:02.084654 15013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:39:02.090085 15013 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 23:39:02.092938 15028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:39:02.094249 15013 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260620 23:39:02.094435 15013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-1/wal
uuid: "acc9e1859c3e45aa9ab8428c27657281"
format_stamp: "Formatted at 2026-06-20 23:38:59 on dist-test-slave-tcgx"
server_key: "b4b7aa72b22d1327b1c18c01f62a07b3"
server_key_iv: "faf10a0724877051055956a791c3da90"
server_key_version: "encryptionkey@0"
I20260620 23:39:02.095664 15013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:39:02.116029 15013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:39:02.120409 15013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:39:02.120716 15013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:39:02.121861 15013 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:39:02.123325 15013 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:39:02.123467 15013 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:39:02.123606 15013 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:39:02.123705 15013 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:39:02.161993 15013 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39353
I20260620 23:39:02.162022 15140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39353 every 8 connection(s)
I20260620 23:39:02.163848 15013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
I20260620 23:39:02.166123 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 15013
I20260620 23:39:02.166754 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:41913
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:41617
--webserver_port=36397
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:39:02 dist-test-slave-tcgx krb5kdc[14240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998742, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:39:02.189221 15011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:02.167926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50083 (local address 127.0.97.62:41617)
0620 23:39:02.168755 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0620 23:39:02.168762 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:39:02.168785 (+ 23us) server_negotiation.cc:407] Connection header received
0620 23:39:02.171468 (+ 2683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:39:02.171475 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:39:02.171599 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:39:02.171707 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:39:02.173890 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:02.174594 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:02.177210 (+ 2616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:02.177650 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:02.177752 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:02.182428 (+ 4676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:39:02.182433 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:39:02.182437 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:39:02.182485 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:39:02.185263 (+ 2778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:39:02.186387 (+ 1124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:39:02.186395 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:39:02.186401 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:39:02.186596 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:39:02.187224 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:39:02.187227 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:39:02.187230 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:39:02.187526 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:39:02.187724 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 23:39:02.188772 (+ 1048us) server_negotiation.cc:299] Negotiation successful
0620 23:39:02.188966 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":698}
I20260620 23:39:02.191606 15143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:02.167341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41617 (local address 127.0.97.2:50083)
0620 23:39:02.168267 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:02.168319 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 23:39:02.171107 (+ 2788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:39:02.172015 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:39:02.172030 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:39:02.172826 (+ 796us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:39:02.173679 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:39:02.173708 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:39:02.174787 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:39:02.174795 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:39:02.177046 (+ 2251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:39:02.177065 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:39:02.177908 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:39:02.177917 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:39:02.178236 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:02.179020 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:39:02.179060 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:39:02.182154 (+ 3094us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:39:02.185463 (+ 3309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:39:02.185476 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:39:02.185500 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:39:02.186216 (+ 716us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:39:02.186744 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:39:02.186752 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:39:02.186757 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:39:02.187088 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:39:02.187716 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:39:02.187728 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:39:02.188298 (+ 570us) client_negotiation.cc:769] Sending connection context
0620 23:39:02.188676 (+ 378us) client_negotiation.cc:240] Negotiation successful
0620 23:39:02.189640 (+ 964us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":353,"threads_started":1}
I20260620 23:39:02.193558 15141 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41617
I20260620 23:39:02.193930 15141 heartbeater.cc:464] Registering TS with master...
I20260620 23:39:02.194725 15141 heartbeater.cc:511] Master 127.0.97.62:41617 requested a full tablet report, sending...
I20260620 23:39:02.196892 14824 ts_manager.cc:194] Registered new tserver with Master: acc9e1859c3e45aa9ab8428c27657281 (127.0.97.2:39353)
I20260620 23:39:02.198046 14824 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:50083
W20260620 23:39:02.522070 15144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:39:02.522365 15144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:39:02.522462 15144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:39:02.529985 15144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:39:02.530107 15144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:39:02.530223 15144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 23:39:02.540258 15144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:41913
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=36397
--tserver_master_addrs=127.0.97.62:41617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:39:02.541570 15144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:39:02.542891 15144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:39:02.556320 15149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:39:02.556293 15152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:39:02.556345 15150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:39:02.557029 15144 server_base.cc:1034] running on GCE node
I20260620 23:39:02.557673 15144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:39:02.558579 15144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:39:02.559780 15144 hybrid_clock.cc:648] HybridClock initialized: now 1781998742559727 us; error 39 us; skew 500 ppm
I20260620 23:39:02.560077 15144 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:39:02 dist-test-slave-tcgx krb5kdc[14240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998742, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:39:02.563238 15144 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 23:39:02.564726 15144 webserver.cc:460] Webserver started at http://127.0.97.3:36397/ using document root <none> and password file <none>
I20260620 23:39:02.565446 15144 fs_manager.cc:359] Metadata directory not provided
I20260620 23:39:02.565589 15144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:39:02.571101 15144 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:39:02.574021 15159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:39:02.575495 15144 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:39:02.575676 15144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-2/wal
uuid: "8a1301fde2de4afe8548772edc8267a9"
format_stamp: "Formatted at 2026-06-20 23:38:59 on dist-test-slave-tcgx"
server_key: "fb6b647b34e634d0ad7fd493cb6f7ed4"
server_key_iv: "0272705e08f39d14596d3236fc873eca"
server_key_version: "encryptionkey@0"
I20260620 23:39:02.576711 15144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:39:02.600308 15144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:39:02.605293 15144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:39:02.605597 15144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:39:02.606626 15144 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:39:02.608269 15144 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:39:02.608410 15144 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:39:02.608546 15144 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:39:02.608632 15144 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:39:02.643431 15144 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41913
I20260620 23:39:02.643501 15271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41913 every 8 connection(s)
I20260620 23:39:02.645054 15144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
I20260620 23:39:02.645792 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 15144
Jun 20 23:39:02 dist-test-slave-tcgx krb5kdc[14240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998742, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:39:02.662883 15011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:02.647028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36778 (local address 127.0.97.62:41617)
0620 23:39:02.647273 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 23:39:02.647282 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:39:02.647737 (+ 455us) server_negotiation.cc:407] Connection header received
0620 23:39:02.647926 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:39:02.647931 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:39:02.648024 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:39:02.648130 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:39:02.649802 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:02.650965 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:02.652471 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:02.653125 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:02.653264 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:02.656445 (+ 3181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:39:02.656452 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:39:02.656456 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:39:02.656499 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:39:02.660065 (+ 3566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:39:02.660788 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:39:02.660796 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:39:02.660800 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:39:02.660972 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:39:02.661348 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:39:02.661353 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:39:02.661359 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:39:02.661651 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:39:02.661966 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0620 23:39:02.662286 (+ 320us) server_negotiation.cc:299] Negotiation successful
0620 23:39:02.662530 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 23:39:02.669492 15276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:02.649234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50089 (local address 127.0.97.62:41617)
0620 23:39:02.649684 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0620 23:39:02.649692 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:39:02.649761 (+ 69us) server_negotiation.cc:407] Connection header received
0620 23:39:02.652406 (+ 2645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:39:02.652414 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:39:02.652509 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:39:02.652611 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:39:02.654591 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:02.655861 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:02.658177 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:02.658622 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:02.658723 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:02.662643 (+ 3920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:39:02.662649 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:39:02.662656 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:39:02.662716 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:39:02.666181 (+ 3465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:39:02.667014 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:39:02.667020 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:39:02.667026 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:39:02.667152 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:39:02.667618 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:39:02.667623 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:39:02.667626 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:39:02.667928 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:39:02.668083 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 23:39:02.668833 (+ 750us) server_negotiation.cc:299] Negotiation successful
0620 23:39:02.669073 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":929,"threads_started":1}
I20260620 23:39:02.670496 15275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:02.648798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41617 (local address 127.0.97.3:50089)
0620 23:39:02.649593 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:02.649655 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 23:39:02.652134 (+ 2479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:39:02.652917 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:39:02.652929 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:39:02.653654 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:39:02.654397 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:39:02.654419 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:39:02.656047 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:39:02.656051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:39:02.658018 (+ 1967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:39:02.658036 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:39:02.658883 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:39:02.658891 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:39:02.659232 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:02.659877 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:39:02.659921 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:39:02.662406 (+ 2485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:39:02.666343 (+ 3937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:39:02.666353 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:39:02.666373 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:39:02.666863 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:39:02.667273 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:39:02.667278 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:39:02.667282 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:39:02.667495 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:39:02.668078 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:39:02.668087 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:39:02.668465 (+ 378us) client_negotiation.cc:769] Sending connection context
0620 23:39:02.668744 (+ 279us) client_negotiation.cc:240] Negotiation successful
0620 23:39:02.669320 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":484,"threads_started":1}
I20260620 23:39:02.672070 15272 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41617
I20260620 23:39:02.672415 15272 heartbeater.cc:464] Registering TS with master...
I20260620 23:39:02.673189 15272 heartbeater.cc:511] Master 127.0.97.62:41617 requested a full tablet report, sending...
I20260620 23:39:02.674780 14824 ts_manager.cc:194] Registered new tserver with Master: 8a1301fde2de4afe8548772edc8267a9 (127.0.97.3:41913)
I20260620 23:39:02.675663 14824 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:50089
I20260620 23:39:02.678522 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:39:02.685674 15276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:02.679470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36782 (local address 127.0.97.62:41617)
0620 23:39:02.679633 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 23:39:02.679637 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:39:02.680145 (+ 508us) server_negotiation.cc:407] Connection header received
0620 23:39:02.680416 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:39:02.680420 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:39:02.680629 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:39:02.680780 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:39:02.681236 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:02.682010 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:02.683514 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:02.683922 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:02.684052 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:02.684570 (+ 518us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:39:02.685091 (+ 521us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:39:02.685267 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 23:39:02.685318 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 23:39:02.685400 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 23:39:02.686630 14824 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:36782:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:39:02.694058 14824 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:39:02.732282 15282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:02.712378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39353 (local address 127.0.0.1:46022)
0620 23:39:02.723901 (+ 11523us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:02.723921 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:39:02.724059 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:39:02.724906 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:39:02.724913 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:39:02.724940 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:39:02.725447 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:39:02.725458 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:39:02.726833 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:39:02.726838 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:39:02.729770 (+ 2932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:39:02.729783 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:39:02.731165 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:39:02.731172 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:39:02.731364 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:02.731415 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 23:39:02.731572 (+ 157us) client_negotiation.cc:240] Negotiation successful
0620 23:39:02.731793 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11382,"mutex_wait_us":123,"thread_start_us":11665,"threads_started":1}
I20260620 23:39:02.733057 15284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:02.707348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46022 (local address 127.0.97.2:39353)
0620 23:39:02.716471 (+ 9123us) server_negotiation.cc:206] Beginning negotiation
0620 23:39:02.716482 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:39:02.724060 (+ 7578us) server_negotiation.cc:407] Connection header received
0620 23:39:02.724220 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:39:02.724227 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:39:02.724474 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:39:02.724669 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:39:02.725601 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:02.726637 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:02.729920 (+ 3283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:02.730981 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:02.731116 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:02.731331 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 23:39:02.732566 (+ 1235us) server_negotiation.cc:299] Negotiation successful
0620 23:39:02.732795 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8947,"thread_start_us":8829,"threads_started":1}
I20260620 23:39:02.733162 15286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:02.708202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33280 (local address 127.0.97.3:41913)
0620 23:39:02.720593 (+ 12391us) server_negotiation.cc:206] Beginning negotiation
0620 23:39:02.720606 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 23:39:02.723895 (+ 3289us) server_negotiation.cc:407] Connection header received
0620 23:39:02.724132 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:39:02.724142 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:39:02.724363 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:39:02.724576 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:39:02.725469 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:02.726983 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:02.729453 (+ 2470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:02.730982 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:02.731117 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:02.732572 (+ 1455us) server_negotiation.cc:1035] Waiting for connection context
0620 23:39:02.732665 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 23:39:02.732832 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12109,"thread_start_us":12411,"threads_started":1}
I20260620 23:39:02.733225 15283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:02.707216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48466 (local address 127.0.97.1:45841)
0620 23:39:02.722321 (+ 15105us) server_negotiation.cc:206] Beginning negotiation
0620 23:39:02.722328 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:39:02.722356 (+ 28us) server_negotiation.cc:407] Connection header received
0620 23:39:02.722454 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:39:02.722479 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:39:02.722698 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:39:02.722987 (+ 289us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:39:02.723992 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:02.726441 (+ 2449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:02.729752 (+ 3311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:02.731252 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:02.731374 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:02.731609 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0620 23:39:02.732745 (+ 1136us) server_negotiation.cc:299] Negotiation successful
0620 23:39:02.732897 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14953,"thread_start_us":16004,"threads_started":1}
I20260620 23:39:02.733599 15287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:02.708062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41913 (local address 127.0.0.1:33280)
0620 23:39:02.723762 (+ 15700us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:02.723784 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 23:39:02.723938 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:39:02.724553 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:39:02.724560 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:39:02.724591 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:39:02.725240 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:39:02.725259 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:39:02.727134 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:39:02.727140 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:39:02.729292 (+ 2152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:39:02.729305 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:39:02.731594 (+ 2289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:39:02.731601 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:39:02.731823 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:02.731856 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 23:39:02.731944 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 23:39:02.732096 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15530,"mutex_wait_us":26,"thread_start_us":8351,"threads_started":1}
I20260620 23:39:02.734010 15285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:02.706867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45841 (local address 127.0.0.1:48466)
0620 23:39:02.717028 (+ 10161us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:02.717090 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 23:39:02.717352 (+ 262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:39:02.723020 (+ 5668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:39:02.723029 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:39:02.723070 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:39:02.723606 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:39:02.723628 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:39:02.726609 (+ 2981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:39:02.726614 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:39:02.729584 (+ 2970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:39:02.729598 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:39:02.731802 (+ 2204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:39:02.731808 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:39:02.731984 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:02.732013 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 23:39:02.732076 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 23:39:02.732191 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9769,"mutex_wait_us":189,"thread_start_us":16725,"threads_started":1}
I20260620 23:39:02.734975 15009 heartbeater.cc:503] Master 127.0.97.62:41617 was elected leader, sending a full tablet report...
I20260620 23:39:02.737361 15076 tablet_service.cc:1467] Processing CreateTablet for tablet 8f0f7c86df2147a993cff5a4f8cbf5ca (DEFAULT_TABLE table=test-table [id=e3c163c54efd48d1af1d1f438a3936ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:39:02.737756 14944 tablet_service.cc:1467] Processing CreateTablet for tablet 8f0f7c86df2147a993cff5a4f8cbf5ca (DEFAULT_TABLE table=test-table [id=e3c163c54efd48d1af1d1f438a3936ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:39:02.738901 15076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f0f7c86df2147a993cff5a4f8cbf5ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:39:02.739102 14944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f0f7c86df2147a993cff5a4f8cbf5ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:39:02.746029 15207 tablet_service.cc:1467] Processing CreateTablet for tablet 8f0f7c86df2147a993cff5a4f8cbf5ca (DEFAULT_TABLE table=test-table [id=e3c163c54efd48d1af1d1f438a3936ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:39:02.747690 15207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f0f7c86df2147a993cff5a4f8cbf5ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:39:02.756417 15288 tablet_bootstrap.cc:492] T 8f0f7c86df2147a993cff5a4f8cbf5ca P a55a5d4f8a194f52b6da9351fdc8bdb1: Bootstrap starting.
I20260620 23:39:02.761332 15288 tablet_bootstrap.cc:654] T 8f0f7c86df2147a993cff5a4f8cbf5ca P a55a5d4f8a194f52b6da9351fdc8bdb1: Neither blocks nor log segments found. Creating new log.
I20260620 23:39:02.762506 15289 tablet_bootstrap.cc:492] T 8f0f7c86df2147a993cff5a4f8cbf5ca P acc9e1859c3e45aa9ab8428c27657281: Bootstrap starting.
I20260620 23:39:02.762830 15288 log.cc:824] T 8f0f7c86df2147a993cff5a4f8cbf5ca P a55a5d4f8a194f52b6da9351fdc8bdb1: Log is configured to *not* fsync() on all Append() calls
I20260620 23:39:02.763379 15290 tablet_bootstrap.cc:492] T 8f0f7c86df2147a993cff5a4f8cbf5ca P 8a1301fde2de4afe8548772edc8267a9: Bootstrap starting.
I20260620 23:39:02.766430 15289 tablet_bootstrap.cc:654] T 8f0f7c86df2147a993cff5a4f8cbf5ca P acc9e1859c3e45aa9ab8428c27657281: Neither blocks nor log segments found. Creating new log.
I20260620 23:39:02.766788 15290 tablet_bootstrap.cc:654] T 8f0f7c86df2147a993cff5a4f8cbf5ca P 8a1301fde2de4afe8548772edc8267a9: Neither blocks nor log segments found. Creating new log.
I20260620 23:39:02.767593 15289 log.cc:824] T 8f0f7c86df2147a993cff5a4f8cbf5ca P acc9e1859c3e45aa9ab8428c27657281: Log is configured to *not* fsync() on all Append() calls
I20260620 23:39:02.768091 15288 tablet_bootstrap.cc:492] T 8f0f7c86df2147a993cff5a4f8cbf5ca P a55a5d4f8a194f52b6da9351fdc8bdb1: No bootstrap required, opened a new log
I20260620 23:39:02.768042 15290 log.cc:824] T 8f0f7c86df2147a993cff5a4f8cbf5ca P 8a1301fde2de4afe8548772edc8267a9: Log is configured to *not* fsync() on all Append() calls
I20260620 23:39:02.768435 15288 ts_tablet_manager.cc:1397] T 8f0f7c86df2147a993cff5a4f8cbf5ca P a55a5d4f8a194f52b6da9351fdc8bdb1: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20260620 23:39:02.770205 15289 tablet_bootstrap.cc:492] T 8f0f7c86df2147a993cff5a4f8cbf5ca P acc9e1859c3e45aa9ab8428c27657281: No bootstrap required, opened a new log
I20260620 23:39:02.770474 15289 ts_tablet_manager.cc:1397] T 8f0f7c86df2147a993cff5a4f8cbf5ca P acc9e1859c3e45aa9ab8428c27657281: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 23:39:02.773311 15290 tablet_bootstrap.cc:492] T 8f0f7c86df2147a993cff5a4f8cbf5ca P 8a1301fde2de4afe8548772edc8267a9: No bootstrap required, opened a new log
I20260620 23:39:02.773602 15290 ts_tablet_manager.cc:1397] T 8f0f7c86df2147a993cff5a4f8cbf5ca P 8a1301fde2de4afe8548772edc8267a9: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260620 23:39:02.778573 15289 raft_consensus.cc:348] T 8f0f7c86df2147a993cff5a4f8cbf5ca P acc9e1859c3e45aa9ab8428c27657281 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55a5d4f8a194f52b6da9351fdc8bdb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45841 } } peers { permanent_uuid: "acc9e1859c3e45aa9ab8428c27657281" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39353 } } peers { permanent_uuid: "8a1301fde2de4afe8548772edc8267a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41913 } }
I20260620 23:39:02.779037 15289 raft_consensus.cc:374] T 8f0f7c86df2147a993cff5a4f8cbf5ca P acc9e1859c3e45aa9ab8428c27657281 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:39:02.778779 15288 raft_consensus.cc:348] T 8f0f7c86df2147a993cff5a4f8cbf5ca P a55a5d4f8a194f52b6da9351fdc8bdb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55a5d4f8a194f52b6da9351fdc8bdb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45841 } } peers { permanent_uuid: "acc9e1859c3e45aa9ab8428c27657281" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39353 } } peers { permanent_uuid: "8a1301fde2de4afe8548772edc8267a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41913 } }
I20260620 23:39:02.779158 15289 raft_consensus.cc:729] T 8f0f7c86df2147a993cff5a4f8cbf5ca P acc9e1859c3e45aa9ab8428c27657281 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc9e1859c3e45aa9ab8428c27657281, State: Initialized, Role: FOLLOWER
I20260620 23:39:02.779282 15288 raft_consensus.cc:374] T 8f0f7c86df2147a993cff5a4f8cbf5ca P a55a5d4f8a194f52b6da9351fdc8bdb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:39:02.779386 15288 raft_consensus.cc:729] T 8f0f7c86df2147a993cff5a4f8cbf5ca P a55a5d4f8a194f52b6da9351fdc8bdb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a55a5d4f8a194f52b6da9351fdc8bdb1, State: Initialized, Role: FOLLOWER
I20260620 23:39:02.779889 15289 consensus_queue.cc:260] T 8f0f7c86df2147a993cff5a4f8cbf5ca P acc9e1859c3e45aa9ab8428c27657281 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55a5d4f8a194f52b6da9351fdc8bdb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45841 } } peers { permanent_uuid: "acc9e1859c3e45aa9ab8428c27657281" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39353 } } peers { permanent_uuid: "8a1301fde2de4afe8548772edc8267a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41913 } }
I20260620 23:39:02.780045 15288 consensus_queue.cc:260] T 8f0f7c86df2147a993cff5a4f8cbf5ca P a55a5d4f8a194f52b6da9351fdc8bdb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55a5d4f8a194f52b6da9351fdc8bdb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45841 } } peers { permanent_uuid: "acc9e1859c3e45aa9ab8428c27657281" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39353 } } peers { permanent_uuid: "8a1301fde2de4afe8548772edc8267a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41913 } }
I20260620 23:39:02.781268 15141 heartbeater.cc:503] Master 127.0.97.62:41617 was elected leader, sending a full tablet report...
I20260620 23:39:02.782132 15289 ts_tablet_manager.cc:1428] T 8f0f7c86df2147a993cff5a4f8cbf5ca P acc9e1859c3e45aa9ab8428c27657281: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260620 23:39:02.784063 15288 ts_tablet_manager.cc:1428] T 8f0f7c86df2147a993cff5a4f8cbf5ca P a55a5d4f8a194f52b6da9351fdc8bdb1: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260620 23:39:02.784734 15290 raft_consensus.cc:348] T 8f0f7c86df2147a993cff5a4f8cbf5ca P 8a1301fde2de4afe8548772edc8267a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55a5d4f8a194f52b6da9351fdc8bdb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45841 } } peers { permanent_uuid: "acc9e1859c3e45aa9ab8428c27657281" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39353 } } peers { permanent_uuid: "8a1301fde2de4afe8548772edc8267a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41913 } }
I20260620 23:39:02.785248 15290 raft_consensus.cc:374] T 8f0f7c86df2147a993cff5a4f8cbf5ca P 8a1301fde2de4afe8548772edc8267a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:39:02.785413 15290 raft_consensus.cc:729] T 8f0f7c86df2147a993cff5a4f8cbf5ca P 8a1301fde2de4afe8548772edc8267a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a1301fde2de4afe8548772edc8267a9, State: Initialized, Role: FOLLOWER
I20260620 23:39:02.786211 15290 consensus_queue.cc:260] T 8f0f7c86df2147a993cff5a4f8cbf5ca P 8a1301fde2de4afe8548772edc8267a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55a5d4f8a194f52b6da9351fdc8bdb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45841 } } peers { permanent_uuid: "acc9e1859c3e45aa9ab8428c27657281" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39353 } } peers { permanent_uuid: "8a1301fde2de4afe8548772edc8267a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41913 } }
I20260620 23:39:02.787570 15272 heartbeater.cc:503] Master 127.0.97.62:41617 was elected leader, sending a full tablet report...
I20260620 23:39:02.789637 15290 ts_tablet_manager.cc:1428] T 8f0f7c86df2147a993cff5a4f8cbf5ca P 8a1301fde2de4afe8548772edc8267a9: Time spent starting tablet: real 0.016s user 0.009s sys 0.007s
I20260620 23:39:02.883816 15294 raft_consensus.cc:482] T 8f0f7c86df2147a993cff5a4f8cbf5ca P acc9e1859c3e45aa9ab8428c27657281 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:39:02.884107 15294 raft_consensus.cc:504] T 8f0f7c86df2147a993cff5a4f8cbf5ca P acc9e1859c3e45aa9ab8428c27657281 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55a5d4f8a194f52b6da9351fdc8bdb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45841 } } peers { permanent_uuid: "acc9e1859c3e45aa9ab8428c27657281" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39353 } } peers { permanent_uuid: "8a1301fde2de4afe8548772edc8267a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41913 } }
I20260620 23:39:02.885934 15294 leader_election.cc:288] T 8f0f7c86df2147a993cff5a4f8cbf5ca P acc9e1859c3e45aa9ab8428c27657281 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a55a5d4f8a194f52b6da9351fdc8bdb1 (127.0.97.1:45841), 8a1301fde2de4afe8548772edc8267a9 (127.0.97.3:41913)
I20260620 23:39:02.892642 15283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:02.886308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41561 (local address 127.0.97.1:45841)
0620 23:39:02.886580 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0620 23:39:02.886589 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:39:02.887273 (+ 684us) server_negotiation.cc:407] Connection header received
0620 23:39:02.887523 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:39:02.887535 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:39:02.887595 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:39:02.887691 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:39:02.888278 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:02.889006 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:02.890820 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:02.891658 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:02.891818 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:02.891933 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 23:39:02.892246 (+ 313us) server_negotiation.cc:299] Negotiation successful
0620 23:39:02.892372 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 23:39:02.892679 15286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:02.886423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57093 (local address 127.0.97.3:41913)
0620 23:39:02.886691 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0620 23:39:02.886698 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:39:02.887346 (+ 648us) server_negotiation.cc:407] Connection header received
0620 23:39:02.887548 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:39:02.887552 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:39:02.887613 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:39:02.887711 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:39:02.888347 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:02.889044 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:02.890888 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:02.891733 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:02.891884 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:02.891993 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 23:39:02.892287 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 23:39:02.892424 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260620 23:39:02.892793 15297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:02.886123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45841 (local address 127.0.97.2:41561)
0620 23:39:02.887147 (+ 1024us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:02.887165 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:39:02.887353 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:39:02.887710 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:39:02.887715 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:39:02.887737 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:39:02.888130 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:39:02.888142 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:39:02.889140 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:39:02.889146 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:39:02.890667 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:39:02.890683 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:39:02.891909 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:39:02.891916 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:39:02.892085 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:02.892109 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 23:39:02.892220 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 23:39:02.892360 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":870,"thread_start_us":531,"threads_started":1}
I20260620 23:39:02.892839 15298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:02.886358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41913 (local address 127.0.97.2:57093)
0620 23:39:02.887255 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:02.887273 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:39:02.887383 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:39:02.887723 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:39:02.887728 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:39:02.887749 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:39:02.888178 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:39:02.888187 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:39:02.889178 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:39:02.889185 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:39:02.890724 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:39:02.890736 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:39:02.891961 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:39:02.891966 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:39:02.892130 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:02.892154 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 23:39:02.892260 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 23:39:02.892386 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":773,"mutex_wait_us":49,"thread_start_us":586,"threads_started":1}
I20260620 23:39:02.893870 15227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f7c86df2147a993cff5a4f8cbf5ca" candidate_uuid: "acc9e1859c3e45aa9ab8428c27657281" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a1301fde2de4afe8548772edc8267a9" is_pre_election: true
I20260620 23:39:02.894160 14964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f7c86df2147a993cff5a4f8cbf5ca" candidate_uuid: "acc9e1859c3e45aa9ab8428c27657281" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a55a5d4f8a194f52b6da9351fdc8bdb1" is_pre_election: true
I20260620 23:39:02.894521 15227 raft_consensus.cc:2454] T 8f0f7c86df2147a993cff5a4f8cbf5ca P 8a1301fde2de4afe8548772edc8267a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acc9e1859c3e45aa9ab8428c27657281 in term 0.
I20260620 23:39:02.894660 14964 raft_consensus.cc:2454] T 8f0f7c86df2147a993cff5a4f8cbf5ca P a55a5d4f8a194f52b6da9351fdc8bdb1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acc9e1859c3e45aa9ab8428c27657281 in term 0.
I20260620 23:39:02.895287 15031 leader_election.cc:302] T 8f0f7c86df2147a993cff5a4f8cbf5ca P acc9e1859c3e45aa9ab8428c27657281 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 8a1301fde2de4afe8548772edc8267a9, a55a5d4f8a194f52b6da9351fdc8bdb1, acc9e1859c3e45aa9ab8428c27657281; no voters:
I20260620 23:39:02.895731 15294 raft_consensus.cc:2781] T 8f0f7c86df2147a993cff5a4f8cbf5ca P acc9e1859c3e45aa9ab8428c27657281 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:39:02.895934 15294 raft_consensus.cc:482] T 8f0f7c86df2147a993cff5a4f8cbf5ca P acc9e1859c3e45aa9ab8428c27657281 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:39:02.896088 15294 raft_consensus.cc:3037] T 8f0f7c86df2147a993cff5a4f8cbf5ca P acc9e1859c3e45aa9ab8428c27657281 [term 0 FOLLOWER]: Advancing to term 1
W20260620 23:39:02.896821 15273 tablet.cc:2369] T 8f0f7c86df2147a993cff5a4f8cbf5ca P 8a1301fde2de4afe8548772edc8267a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:39:02.898411 15294 raft_consensus.cc:504] T 8f0f7c86df2147a993cff5a4f8cbf5ca P acc9e1859c3e45aa9ab8428c27657281 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55a5d4f8a194f52b6da9351fdc8bdb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45841 } } peers { permanent_uuid: "acc9e1859c3e45aa9ab8428c27657281" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39353 } } peers { permanent_uuid: "8a1301fde2de4afe8548772edc8267a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41913 } }
I20260620 23:39:02.899147 15294 leader_election.cc:288] T 8f0f7c86df2147a993cff5a4f8cbf5ca P acc9e1859c3e45aa9ab8428c27657281 [CANDIDATE]: Term 1 election: Requested vote from peers a55a5d4f8a194f52b6da9351fdc8bdb1 (127.0.97.1:45841), 8a1301fde2de4afe8548772edc8267a9 (127.0.97.3:41913)
I20260620 23:39:02.899653 14964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f7c86df2147a993cff5a4f8cbf5ca" candidate_uuid: "acc9e1859c3e45aa9ab8428c27657281" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a55a5d4f8a194f52b6da9351fdc8bdb1"
I20260620 23:39:02.899673 15227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f0f7c86df2147a993cff5a4f8cbf5ca" candidate_uuid: "acc9e1859c3e45aa9ab8428c27657281" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a1301fde2de4afe8548772edc8267a9"
I20260620 23:39:02.899887 14964 raft_consensus.cc:3037] T 8f0f7c86df2147a993cff5a4f8cbf5ca P a55a5d4f8a194f52b6da9351fdc8bdb1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:39:02.899886 15227 raft_consensus.cc:3037] T 8f0f7c86df2147a993cff5a4f8cbf5ca P 8a1301fde2de4afe8548772edc8267a9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:39:02.902304 14964 raft_consensus.cc:2454] T 8f0f7c86df2147a993cff5a4f8cbf5ca P a55a5d4f8a194f52b6da9351fdc8bdb1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acc9e1859c3e45aa9ab8428c27657281 in term 1.
I20260620 23:39:02.902724 15227 raft_consensus.cc:2454] T 8f0f7c86df2147a993cff5a4f8cbf5ca P 8a1301fde2de4afe8548772edc8267a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acc9e1859c3e45aa9ab8428c27657281 in term 1.
I20260620 23:39:02.902743 15032 leader_election.cc:302] T 8f0f7c86df2147a993cff5a4f8cbf5ca P acc9e1859c3e45aa9ab8428c27657281 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a55a5d4f8a194f52b6da9351fdc8bdb1, acc9e1859c3e45aa9ab8428c27657281; no voters:
I20260620 23:39:02.903146 15294 raft_consensus.cc:2781] T 8f0f7c86df2147a993cff5a4f8cbf5ca P acc9e1859c3e45aa9ab8428c27657281 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:39:02.903609 15294 raft_consensus.cc:686] T 8f0f7c86df2147a993cff5a4f8cbf5ca P acc9e1859c3e45aa9ab8428c27657281 [term 1 LEADER]: Becoming Leader. State: Replica: acc9e1859c3e45aa9ab8428c27657281, State: Running, Role: LEADER
I20260620 23:39:02.904111 15294 consensus_queue.cc:237] T 8f0f7c86df2147a993cff5a4f8cbf5ca P acc9e1859c3e45aa9ab8428c27657281 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55a5d4f8a194f52b6da9351fdc8bdb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45841 } } peers { permanent_uuid: "acc9e1859c3e45aa9ab8428c27657281" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39353 } } peers { permanent_uuid: "8a1301fde2de4afe8548772edc8267a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41913 } }
I20260620 23:39:02.909085 14824 catalog_manager.cc:5495] T 8f0f7c86df2147a993cff5a4f8cbf5ca P acc9e1859c3e45aa9ab8428c27657281 reported cstate change: term changed from 0 to 1, leader changed from <none> to acc9e1859c3e45aa9ab8428c27657281 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "acc9e1859c3e45aa9ab8428c27657281" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a55a5d4f8a194f52b6da9351fdc8bdb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acc9e1859c3e45aa9ab8428c27657281" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a1301fde2de4afe8548772edc8267a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41913 } health_report { overall_health: UNKNOWN } } }
W20260620 23:39:02.915970 15142 tablet.cc:2369] T 8f0f7c86df2147a993cff5a4f8cbf5ca P acc9e1859c3e45aa9ab8428c27657281: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:39:02.929697 15284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:02.924503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46032 (local address 127.0.97.2:39353)
0620 23:39:02.924682 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 23:39:02.924686 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:39:02.924703 (+ 17us) server_negotiation.cc:407] Connection header received
0620 23:39:02.924795 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:39:02.924799 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:39:02.924875 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:39:02.925002 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:39:02.925350 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:02.926105 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:02.927483 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:02.927924 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:02.928021 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:02.928516 (+ 495us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:39:02.929072 (+ 556us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:39:02.929190 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 23:39:02.929289 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 23:39:02.929370 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 23:39:02.938771 14964 raft_consensus.cc:1261] T 8f0f7c86df2147a993cff5a4f8cbf5ca P a55a5d4f8a194f52b6da9351fdc8bdb1 [term 1 FOLLOWER]: Refusing update from remote peer acc9e1859c3e45aa9ab8428c27657281: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:39:02.938771 15227 raft_consensus.cc:1261] T 8f0f7c86df2147a993cff5a4f8cbf5ca P 8a1301fde2de4afe8548772edc8267a9 [term 1 FOLLOWER]: Refusing update from remote peer acc9e1859c3e45aa9ab8428c27657281: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:39:02.940068 15294 consensus_queue.cc:1035] T 8f0f7c86df2147a993cff5a4f8cbf5ca P acc9e1859c3e45aa9ab8428c27657281 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a1301fde2de4afe8548772edc8267a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:39:02.940452 15299 consensus_queue.cc:1035] T 8f0f7c86df2147a993cff5a4f8cbf5ca P acc9e1859c3e45aa9ab8428c27657281 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a55a5d4f8a194f52b6da9351fdc8bdb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 23:39:02.941525 15010 tablet.cc:2369] T 8f0f7c86df2147a993cff5a4f8cbf5ca P a55a5d4f8a194f52b6da9351fdc8bdb1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:39:02.952913 15302 mvcc.cc:205] Tried to move back new op lower bound from 7299066851065217024 to 7299066850939392000. Current Snapshot: MvccSnapshot[applied={T|T < 7299066851065217024}]
I20260620 23:39:02.967590 14823 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:36782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:39:02.967926 14823 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:36782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:39:02.971747 14823 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f7e7ba8cc23b4fdbaf0a02f1a75e37b9: Sending DeleteTablet for 3 replicas of tablet 8f0f7c86df2147a993cff5a4f8cbf5ca
I20260620 23:39:02.973145 14944 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f0f7c86df2147a993cff5a4f8cbf5ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:39:02 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48466
I20260620 23:39:02.973151 15076 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f0f7c86df2147a993cff5a4f8cbf5ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:39:02 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46022
I20260620 23:39:02.974133 15207 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f0f7c86df2147a993cff5a4f8cbf5ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:39:02 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33280
I20260620 23:39:02.974604 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 14861
I20260620 23:39:02.981904 15313 tablet_replica.cc:330] T 8f0f7c86df2147a993cff5a4f8cbf5ca P acc9e1859c3e45aa9ab8428c27657281: stopping tablet replica
I20260620 23:39:02.983233 15315 tablet_replica.cc:330] T 8f0f7c86df2147a993cff5a4f8cbf5ca P 8a1301fde2de4afe8548772edc8267a9: stopping tablet replica
I20260620 23:39:02.983733 15315 raft_consensus.cc:2229] T 8f0f7c86df2147a993cff5a4f8cbf5ca P 8a1301fde2de4afe8548772edc8267a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:39:02.984277 15315 raft_consensus.cc:2258] T 8f0f7c86df2147a993cff5a4f8cbf5ca P 8a1301fde2de4afe8548772edc8267a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:39:02.984962 15313 raft_consensus.cc:2229] T 8f0f7c86df2147a993cff5a4f8cbf5ca P acc9e1859c3e45aa9ab8428c27657281 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:39:02.985672 15313 raft_consensus.cc:2258] T 8f0f7c86df2147a993cff5a4f8cbf5ca P acc9e1859c3e45aa9ab8428c27657281 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:39:02.986474 15314 tablet_replica.cc:330] T 8f0f7c86df2147a993cff5a4f8cbf5ca P a55a5d4f8a194f52b6da9351fdc8bdb1: stopping tablet replica
I20260620 23:39:02.986987 15314 raft_consensus.cc:2229] T 8f0f7c86df2147a993cff5a4f8cbf5ca P a55a5d4f8a194f52b6da9351fdc8bdb1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:39:02.987609 15314 raft_consensus.cc:2258] T 8f0f7c86df2147a993cff5a4f8cbf5ca P a55a5d4f8a194f52b6da9351fdc8bdb1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:39:02.988168 15313 ts_tablet_manager.cc:1905] T 8f0f7c86df2147a993cff5a4f8cbf5ca P acc9e1859c3e45aa9ab8428c27657281: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:39:02.990448 15314 ts_tablet_manager.cc:1905] T 8f0f7c86df2147a993cff5a4f8cbf5ca P a55a5d4f8a194f52b6da9351fdc8bdb1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:39:02.994427 15283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:02.976195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48470 (local address 127.0.97.1:45841)
0620 23:39:02.976353 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 23:39:02.976358 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:39:02.976373 (+ 15us) server_negotiation.cc:407] Connection header received
0620 23:39:02.976423 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:39:02.976426 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:39:02.976506 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:39:02.976626 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:39:02.978457 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:02.979557 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:02.981322 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:02.982027 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:02.982165 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:02.986673 (+ 4508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:39:02.986687 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:39:02.986701 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:39:02.986742 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:39:02.990865 (+ 4123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:39:02.991848 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:39:02.991857 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:39:02.991860 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:39:02.992053 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:39:02.992512 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:39:02.992518 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:39:02.992521 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:39:02.993096 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:39:02.993441 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0620 23:39:02.993773 (+ 332us) server_negotiation.cc:299] Negotiation successful
0620 23:39:02.994014 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 23:39:02.994676 15315 ts_tablet_manager.cc:1905] T 8f0f7c86df2147a993cff5a4f8cbf5ca P 8a1301fde2de4afe8548772edc8267a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:39:02.995415 15004 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:39:02.995796 15313 ts_tablet_manager.cc:1918] T 8f0f7c86df2147a993cff5a4f8cbf5ca P acc9e1859c3e45aa9ab8428c27657281: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:39:02.996124 15313 log.cc:1196] T 8f0f7c86df2147a993cff5a4f8cbf5ca P acc9e1859c3e45aa9ab8428c27657281: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-1/wal/wals/8f0f7c86df2147a993cff5a4f8cbf5ca
I20260620 23:39:02.996941 15313 ts_tablet_manager.cc:1939] T 8f0f7c86df2147a993cff5a4f8cbf5ca P acc9e1859c3e45aa9ab8428c27657281: Deleting consensus metadata
I20260620 23:39:02.999574 14808 catalog_manager.cc:4841] TS acc9e1859c3e45aa9ab8428c27657281 (127.0.97.2:39353): tablet 8f0f7c86df2147a993cff5a4f8cbf5ca (table test-table [id=e3c163c54efd48d1af1d1f438a3936ee]) successfully deleted
I20260620 23:39:03.002538 15315 ts_tablet_manager.cc:1918] T 8f0f7c86df2147a993cff5a4f8cbf5ca P 8a1301fde2de4afe8548772edc8267a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:39:03.003098 15315 log.cc:1196] T 8f0f7c86df2147a993cff5a4f8cbf5ca P 8a1301fde2de4afe8548772edc8267a9: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-2/wal/wals/8f0f7c86df2147a993cff5a4f8cbf5ca
I20260620 23:39:03.003778 15315 ts_tablet_manager.cc:1939] T 8f0f7c86df2147a993cff5a4f8cbf5ca P 8a1301fde2de4afe8548772edc8267a9: Deleting consensus metadata
I20260620 23:39:03.006443 14810 catalog_manager.cc:4841] TS 8a1301fde2de4afe8548772edc8267a9 (127.0.97.3:41913): tablet 8f0f7c86df2147a993cff5a4f8cbf5ca (table test-table [id=e3c163c54efd48d1af1d1f438a3936ee]) successfully deleted
I20260620 23:39:03.093274 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 14861
I20260620 23:39:03.112272 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 15013
W20260620 23:39:03.112305 14811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:45841 (error 108)
I20260620 23:39:03.114406 15285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:03.113907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45841 (local address 127.0.0.1:48476)
0620 23:39:03.114095 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:03.114230 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260620 23:39:03.114689 14811 catalog_manager.cc:4584] TS a55a5d4f8a194f52b6da9351fdc8bdb1 (127.0.97.1:45841): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8f0f7c86df2147a993cff5a4f8cbf5ca: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45841: connect: Connection refused (error 111)
I20260620 23:39:03.128147 15284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:03.113559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46040 (local address 127.0.97.2:39353)
0620 23:39:03.113742 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 23:39:03.113750 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:39:03.113771 (+ 21us) server_negotiation.cc:407] Connection header received
0620 23:39:03.113840 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:39:03.113846 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:39:03.113944 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:39:03.114054 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:39:03.115666 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:03.116477 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:03.117840 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:03.118306 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:03.118458 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:03.121466 (+ 3008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:39:03.121481 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:39:03.121498 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:39:03.121551 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:39:03.124878 (+ 3327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:39:03.125734 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:39:03.125742 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:39:03.125745 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:39:03.125929 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:39:03.126371 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:39:03.126377 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:39:03.126380 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:39:03.126762 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:39:03.127393 (+ 631us) server_negotiation.cc:1035] Waiting for connection context
0620 23:39:03.127558 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 23:39:03.127798 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 23:39:03.129060 15136 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:39:03.168259 15285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:03.167725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45841 (local address 127.0.0.1:48482)
0620 23:39:03.167961 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:03.168073 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260620 23:39:03.169385 15285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:03.168960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45841 (local address 127.0.0.1:48486)
0620 23:39:03.169128 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:03.169221 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 23:39:03.230154 15285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:03.229554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45841 (local address 127.0.0.1:48492)
0620 23:39:03.229807 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:03.229921 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260620 23:39:03.231276 15285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:03.230861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45841 (local address 127.0.0.1:48502)
0620 23:39:03.231063 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:03.231143 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260620 23:39:03.246881 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 15013
I20260620 23:39:03.266438 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 15144
I20260620 23:39:03.279663 15286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:03.267364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33294 (local address 127.0.97.3:41913)
0620 23:39:03.267582 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 23:39:03.267587 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:39:03.267614 (+ 27us) server_negotiation.cc:407] Connection header received
0620 23:39:03.267782 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:39:03.267786 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:39:03.267854 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:39:03.267959 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:39:03.269228 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:03.269970 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:03.271292 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:03.271679 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:03.271770 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:03.274327 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:39:03.274343 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:39:03.274359 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:39:03.274401 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:39:03.277195 (+ 2794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:39:03.277852 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:39:03.277860 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:39:03.277862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:39:03.277999 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:39:03.278299 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:39:03.278302 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:39:03.278305 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:39:03.278657 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:39:03.278888 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0620 23:39:03.279193 (+ 305us) server_negotiation.cc:299] Negotiation successful
0620 23:39:03.279360 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 23:39:03.280468 15267 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:39:03.312597 15285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:03.312019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45841 (local address 127.0.0.1:48506)
0620 23:39:03.312277 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:03.312395 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260620 23:39:03.314157 15285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:03.313565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45841 (local address 127.0.0.1:48518)
0620 23:39:03.313845 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:03.313968 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260620 23:39:03.372181 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 15144
I20260620 23:39:03.389086 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 14791
I20260620 23:39:03.390311 14854 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:39:03.507790 15285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:03.507052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45841 (local address 127.0.0.1:48526)
0620 23:39:03.507417 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:03.507560 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260620 23:39:03.507725 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 14791
I20260620 23:39:03.533355 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41617
--webserver_interface=127.0.97.62
--webserver_port=44471
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41617
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:39:03.874984 15321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:39:03.875332 15321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:39:03.875429 15321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:39:03.882898 15321 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:39:03.883114 15321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:39:03.883214 15321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:39:03.883315 15321 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:39:03.883410 15321 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:39:03.893591 15321 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41617
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=44471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:39:03.895071 15321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:39:03.896471 15321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:39:03.908443 15326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:39:03.908528 15329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:39:03.908609 15327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:39:03.909063 15321 server_base.cc:1034] running on GCE node
I20260620 23:39:03.909844 15321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:39:03.911358 15321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:39:03.912611 15321 hybrid_clock.cc:648] HybridClock initialized: now 1781998743912554 us; error 57 us; skew 500 ppm
I20260620 23:39:03.912985 15321 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:39:03 dist-test-slave-tcgx krb5kdc[14240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998743, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:39:03.916158 15321 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 23:39:03.917658 15321 webserver.cc:460] Webserver started at http://127.0.97.62:44471/ using document root <none> and password file <none>
I20260620 23:39:03.918373 15321 fs_manager.cc:359] Metadata directory not provided
I20260620 23:39:03.918515 15321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:39:03.924121 15321 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 23:39:03.927217 15336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:39:03.928745 15321 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 23:39:03.928957 15321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/master-0/wal
uuid: "f7e7ba8cc23b4fdbaf0a02f1a75e37b9"
format_stamp: "Formatted at 2026-06-20 23:38:57 on dist-test-slave-tcgx"
server_key: "b3b53e6a0c350ff5e38b66f00ea6834e"
server_key_iv: "2503cbea121e0a69915fc15e3a7ce45b"
server_key_version: "encryptionkey@0"
I20260620 23:39:03.930052 15321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:39:03.959988 15321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:39:03.964437 15321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:39:03.964730 15321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:39:03.984004 15321 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41617
I20260620 23:39:03.984025 15387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41617 every 8 connection(s)
I20260620 23:39:03.985620 15321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:39:03.987978 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 15321
I20260620 23:39:03.988971 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:45841
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:41617
--webserver_port=34901
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:39:03.995667 15388 sys_catalog.cc:263] Verifying existing consensus state
I20260620 23:39:04.000695 15388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7e7ba8cc23b4fdbaf0a02f1a75e37b9: Bootstrap starting.
I20260620 23:39:04.021842 15388 log.cc:824] T 00000000000000000000000000000000 P f7e7ba8cc23b4fdbaf0a02f1a75e37b9: Log is configured to *not* fsync() on all Append() calls
I20260620 23:39:04.038017 15388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7e7ba8cc23b4fdbaf0a02f1a75e37b9: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:39:04.038585 15388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7e7ba8cc23b4fdbaf0a02f1a75e37b9: Bootstrap complete.
I20260620 23:39:04.048099 15388 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7e7ba8cc23b4fdbaf0a02f1a75e37b9 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7e7ba8cc23b4fdbaf0a02f1a75e37b9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41617 } }
I20260620 23:39:04.048997 15388 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7e7ba8cc23b4fdbaf0a02f1a75e37b9 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7e7ba8cc23b4fdbaf0a02f1a75e37b9, State: Initialized, Role: FOLLOWER
I20260620 23:39:04.049672 15388 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7e7ba8cc23b4fdbaf0a02f1a75e37b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7e7ba8cc23b4fdbaf0a02f1a75e37b9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41617 } }
I20260620 23:39:04.049911 15388 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7e7ba8cc23b4fdbaf0a02f1a75e37b9 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:39:04.050036 15388 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7e7ba8cc23b4fdbaf0a02f1a75e37b9 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:39:04.050227 15388 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7e7ba8cc23b4fdbaf0a02f1a75e37b9 [term 2 FOLLOWER]: Advancing to term 3
I20260620 23:39:04.054021 15388 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7e7ba8cc23b4fdbaf0a02f1a75e37b9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7e7ba8cc23b4fdbaf0a02f1a75e37b9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41617 } }
I20260620 23:39:04.054533 15388 leader_election.cc:302] T 00000000000000000000000000000000 P f7e7ba8cc23b4fdbaf0a02f1a75e37b9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7e7ba8cc23b4fdbaf0a02f1a75e37b9; no voters:
I20260620 23:39:04.055279 15388 leader_election.cc:288] T 00000000000000000000000000000000 P f7e7ba8cc23b4fdbaf0a02f1a75e37b9 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 23:39:04.055415 15392 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7e7ba8cc23b4fdbaf0a02f1a75e37b9 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 23:39:04.056687 15392 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7e7ba8cc23b4fdbaf0a02f1a75e37b9 [term 3 LEADER]: Becoming Leader. State: Replica: f7e7ba8cc23b4fdbaf0a02f1a75e37b9, State: Running, Role: LEADER
I20260620 23:39:04.057304 15392 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7e7ba8cc23b4fdbaf0a02f1a75e37b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7e7ba8cc23b4fdbaf0a02f1a75e37b9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41617 } }
I20260620 23:39:04.057564 15388 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7e7ba8cc23b4fdbaf0a02f1a75e37b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:39:04.059876 15394 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7e7ba8cc23b4fdbaf0a02f1a75e37b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7e7ba8cc23b4fdbaf0a02f1a75e37b9. Latest consensus state: current_term: 3 leader_uuid: "f7e7ba8cc23b4fdbaf0a02f1a75e37b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7e7ba8cc23b4fdbaf0a02f1a75e37b9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41617 } } }
I20260620 23:39:04.060276 15394 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7e7ba8cc23b4fdbaf0a02f1a75e37b9 [sys.catalog]: This master's current role is: LEADER
I20260620 23:39:04.060881 15393 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7e7ba8cc23b4fdbaf0a02f1a75e37b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "f7e7ba8cc23b4fdbaf0a02f1a75e37b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7e7ba8cc23b4fdbaf0a02f1a75e37b9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41617 } } }
I20260620 23:39:04.061142 15393 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7e7ba8cc23b4fdbaf0a02f1a75e37b9 [sys.catalog]: This master's current role is: LEADER
I20260620 23:39:04.072204 15399 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:39:04.083748 15399 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:39:04.085548 15399 catalog_manager.cc:1259] Loaded cluster ID: 7523bac8126c481e9cfe3986e84b2d5f
I20260620 23:39:04.085762 15399 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 23:39:04.088577 15399 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f5e205409ad google::LogMessage::Fail() at ??:0
@ 0x7f5e2054388a google::LogMessage::SendToLog() at ??:0
@ 0x7f5e205404de google::LogMessage::Flush() at ??:0
@ 0x7f5e20541a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f5e35f73512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f5e35f70976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f5e35ff8610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f5e35ff8439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f5e2318c345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f5e2318f91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f5e2318f799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f5e2316aeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f5e264446db start_thread at ??:0
@ 0x7f5e1f3da61f clone at ??:0
W20260620 23:39:04.413667 15390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:39:04.413990 15390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:39:04.414155 15390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:39:04.421640 15390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:39:04.421768 15390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:39:04.421979 15390 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 23:39:04.432564 15390 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:45841
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=34901
--tserver_master_addrs=127.0.97.62:41617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:39:04.433988 15390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:39:04.435369 15390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:39:04.449740 15432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:39:04.449856 15429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:39:04.449836 15430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:39:04.450606 15390 server_base.cc:1034] running on GCE node
I20260620 23:39:04.451359 15390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:39:04.452373 15390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:39:04.453583 15390 hybrid_clock.cc:648] HybridClock initialized: now 1781998744453532 us; error 52 us; skew 500 ppm
I20260620 23:39:04.453933 15390 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:39:04 dist-test-slave-tcgx krb5kdc[14240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998744, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:39:04.457186 15390 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 23:39:04.458622 15390 webserver.cc:460] Webserver started at http://127.0.97.1:34901/ using document root <none> and password file <none>
I20260620 23:39:04.459365 15390 fs_manager.cc:359] Metadata directory not provided
I20260620 23:39:04.459470 15390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:39:04.465796 15390 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260620 23:39:04.468870 15439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:39:04.470230 15390 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:39:04.470362 15390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-0/wal
uuid: "a55a5d4f8a194f52b6da9351fdc8bdb1"
format_stamp: "Formatted at 2026-06-20 23:38:58 on dist-test-slave-tcgx"
server_key: "1c58a1c8ed30db672d5e9ccb3c142f8e"
server_key_iv: "b1cbda2f23d38799a6c9e3eacaa16b46"
server_key_version: "encryptionkey@0"
I20260620 23:39:04.471361 15390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:39:04.493903 15390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:39:04.498245 15390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:39:04.498473 15390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:39:04.499598 15390 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:39:04.501626 15446 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260620 23:39:04.507423 15446 ts_tablet_manager.cc:1852] T 8f0f7c86df2147a993cff5a4f8cbf5ca P a55a5d4f8a194f52b6da9351fdc8bdb1: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260620 23:39:04.508446 15446 ts_tablet_manager.cc:1905] T 8f0f7c86df2147a993cff5a4f8cbf5ca P a55a5d4f8a194f52b6da9351fdc8bdb1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:39:04.516219 15446 ts_tablet_manager.cc:1918] T 8f0f7c86df2147a993cff5a4f8cbf5ca P a55a5d4f8a194f52b6da9351fdc8bdb1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:39:04.516412 15446 log.cc:1196] T 8f0f7c86df2147a993cff5a4f8cbf5ca P a55a5d4f8a194f52b6da9351fdc8bdb1: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-0/wal/wals/8f0f7c86df2147a993cff5a4f8cbf5ca
I20260620 23:39:04.516928 15446 ts_tablet_manager.cc:1939] T 8f0f7c86df2147a993cff5a4f8cbf5ca P a55a5d4f8a194f52b6da9351fdc8bdb1: Deleting consensus metadata
I20260620 23:39:04.517347 15390 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260620 23:39:04.517454 15390 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.016s user 0.001s sys 0.000s
I20260620 23:39:04.517560 15390 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:39:04.517618 15390 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:39:04.551910 15390 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45841
I20260620 23:39:04.552012 15552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45841 every 8 connection(s)
I20260620 23:39:04.553303 15390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
I20260620 23:39:04.557839 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 15390
I20260620 23:39:04.558609 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:39353
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:41617
--webserver_port=40057
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:39:04.558847 15555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:04.556455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41617 (local address 127.0.97.1:60161)
0620 23:39:04.557241 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:04.557508 (+ 267us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":293,"threads_started":1}
W20260620 23:39:04.559614 15553 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:41617 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:41617: Client connection negotiation failed: client connection to 127.0.97.62:41617: connect: Connection refused (error 111)
I20260620 23:39:04.560650 15555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:04.560178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41617 (local address 127.0.97.1:39763)
0620 23:39:04.560377 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:04.560485 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 23:39:04.561625 15555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:04.561196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41617 (local address 127.0.97.1:51657)
0620 23:39:04.561361 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:04.561461 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 23:39:04.561878 15553 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 23:39:04.894351 15556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:39:04.894654 15556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:39:04.894824 15556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:39:04.902420 15556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:39:04.902562 15556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:39:04.902745 15556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 23:39:04.913944 15556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:39353
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=40057
--tserver_master_addrs=127.0.97.62:41617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:39:04.916119 15556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:39:04.918159 15556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:39:04.935525 15561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:39:04.937844 15562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:39:04.938030 15564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:39:04.939323 15556 server_base.cc:1034] running on GCE node
I20260620 23:39:04.940239 15556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:39:04.941617 15556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:39:04.942885 15556 hybrid_clock.cc:648] HybridClock initialized: now 1781998744942809 us; error 61 us; skew 500 ppm
I20260620 23:39:04.943296 15556 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:39:04 dist-test-slave-tcgx krb5kdc[14240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998744, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:39:04.947934 15556 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 23:39:04.949831 15556 webserver.cc:460] Webserver started at http://127.0.97.2:40057/ using document root <none> and password file <none>
I20260620 23:39:04.950881 15556 fs_manager.cc:359] Metadata directory not provided
I20260620 23:39:04.951048 15556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:39:04.958962 15556 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.002s
I20260620 23:39:04.963050 15571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:39:04.965094 15556 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 23:39:04.965287 15556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-1/wal
uuid: "acc9e1859c3e45aa9ab8428c27657281"
format_stamp: "Formatted at 2026-06-20 23:38:59 on dist-test-slave-tcgx"
server_key: "b4b7aa72b22d1327b1c18c01f62a07b3"
server_key_iv: "faf10a0724877051055956a791c3da90"
server_key_version: "encryptionkey@0"
I20260620 23:39:04.966356 15556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:39:04.985793 15556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:39:04.990608 15556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:39:04.990914 15556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:39:04.991981 15556 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:39:04.993422 15556 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:39:04.993546 15556 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:39:04.993695 15556 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:39:04.993783 15556 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:39:05.028486 15556 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39353
I20260620 23:39:05.028577 15683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39353 every 8 connection(s)
I20260620 23:39:05.030035 15556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
I20260620 23:39:05.033552 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 15556
I20260620 23:39:05.034327 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:41913
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:41617
--webserver_port=36397
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:39:05.035478 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:05.033037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41617 (local address 127.0.97.2:49983)
0620 23:39:05.033897 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:05.034200 (+ 303us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":321,"threads_started":1}
W20260620 23:39:05.036257 15684 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:41617 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:41617: Client connection negotiation failed: client connection to 127.0.97.62:41617: connect: Connection refused (error 111)
I20260620 23:39:05.037261 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:05.036796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41617 (local address 127.0.97.2:52229)
0620 23:39:05.037004 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:05.037100 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 23:39:05.038275 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:05.037818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41617 (local address 127.0.97.2:44057)
0620 23:39:05.037996 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:05.038114 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 23:39:05.038542 15684 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 23:39:05.370779 15687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:39:05.371101 15687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:39:05.371299 15687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:39:05.378778 15687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:39:05.378907 15687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:39:05.379122 15687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 23:39:05.389197 15687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:41913
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=36397
--tserver_master_addrs=127.0.97.62:41617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:39:05.390599 15687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:39:05.391986 15687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:39:05.405804 15693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:39:05.405771 15695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:39:05.405716 15692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:39:05.406423 15687 server_base.cc:1034] running on GCE node
I20260620 23:39:05.407207 15687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:39:05.408226 15687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:39:05.409462 15687 hybrid_clock.cc:648] HybridClock initialized: now 1781998745409406 us; error 53 us; skew 500 ppm
I20260620 23:39:05.409804 15687 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:39:05 dist-test-slave-tcgx krb5kdc[14240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998745, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:39:05.413173 15687 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 23:39:05.414706 15687 webserver.cc:460] Webserver started at http://127.0.97.3:36397/ using document root <none> and password file <none>
I20260620 23:39:05.415448 15687 fs_manager.cc:359] Metadata directory not provided
I20260620 23:39:05.415551 15687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:39:05.421017 15687 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 23:39:05.423803 15702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:39:05.425233 15687 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 23:39:05.425433 15687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-2/wal
uuid: "8a1301fde2de4afe8548772edc8267a9"
format_stamp: "Formatted at 2026-06-20 23:38:59 on dist-test-slave-tcgx"
server_key: "fb6b647b34e634d0ad7fd493cb6f7ed4"
server_key_iv: "0272705e08f39d14596d3236fc873eca"
server_key_version: "encryptionkey@0"
I20260620 23:39:05.426510 15687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:39:05.448577 15687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:39:05.453150 15687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:39:05.453437 15687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:39:05.454452 15687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:39:05.456056 15687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:39:05.456175 15687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:39:05.456310 15687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:39:05.456398 15687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:39:05.490272 15687 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41913
I20260620 23:39:05.490368 15814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41913 every 8 connection(s)
I20260620 23:39:05.491811 15687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
I20260620 23:39:05.496942 15817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:05.494839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41617 (local address 127.0.97.3:36257)
0620 23:39:05.495704 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:05.495924 (+ 220us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":339,"threads_started":1}
W20260620 23:39:05.497645 15815 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:41617 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:41617: Client connection negotiation failed: client connection to 127.0.97.62:41617: connect: Connection refused (error 111)
I20260620 23:39:05.498356 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 15687
I20260620 23:39:05.498607 15817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:05.498162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41617 (local address 127.0.97.3:58707)
0620 23:39:05.498365 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:05.498479 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 23:39:05.499648 15817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:05.499235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41617 (local address 127.0.97.3:52023)
0620 23:39:05.499384 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:05.499488 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 23:39:05.499910 15815 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 23:39:05.563345 15819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:05.562456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41617 (local address 127.0.97.1:43031)
0620 23:39:05.562990 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:05.563107 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":220,"threads_started":1}
I20260620 23:39:06.040179 15820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:06.039230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41617 (local address 127.0.97.2:40087)
0620 23:39:06.039807 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:06.039943 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":289,"threads_started":1}
I20260620 23:39:06.501325 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:06.500544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41617 (local address 127.0.97.3:46329)
0620 23:39:06.500984 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:06.501070 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":216,"threads_started":1}
I20260620 23:39:06.565032 15822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:06.564321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41617 (local address 127.0.97.1:37983)
0620 23:39:06.564785 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:06.564872 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":249,"threads_started":1}
I20260620 23:39:07.041935 15823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:07.041179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41617 (local address 127.0.97.2:46331)
0620 23:39:07.041681 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:07.041770 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":266,"threads_started":1}
I20260620 23:39:07.503003 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:07.502213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41617 (local address 127.0.97.3:42309)
0620 23:39:07.502694 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:07.502775 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":240,"threads_started":1}
I20260620 23:39:07.567037 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:07.566108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41617 (local address 127.0.97.1:34113)
0620 23:39:07.566646 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:07.566767 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":288,"threads_started":1}
I20260620 23:39:08.043694 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:08.042909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41617 (local address 127.0.97.2:47915)
0620 23:39:08.043419 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:08.043518 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":245,"threads_started":1}
I20260620 23:39:08.504680 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:08.503919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41617 (local address 127.0.97.3:41197)
0620 23:39:08.504414 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:08.504501 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":264,"threads_started":1}
I20260620 23:39:08.568747 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:08.568040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41617 (local address 127.0.97.1:50905)
0620 23:39:08.568490 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:08.568579 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":224,"threads_started":1}
I20260620 23:39:09.045257 15829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:09.044553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41617 (local address 127.0.97.2:49351)
0620 23:39:09.045004 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:09.045083 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":220,"threads_started":1}
I20260620 23:39:09.506875 15830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:09.505973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41617 (local address 127.0.97.3:52579)
0620 23:39:09.506562 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:09.506652 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":321,"threads_started":1}
I20260620 23:39:09.570777 15831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:09.569846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41617 (local address 127.0.97.1:51149)
0620 23:39:09.570472 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:09.570568 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":328,"threads_started":1}
I20260620 23:39:10.047171 15832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:10.046348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41617 (local address 127.0.97.2:51205)
0620 23:39:10.046809 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:10.046989 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":236,"threads_started":1}
I20260620 23:39:10.508730 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:10.507933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41617 (local address 127.0.97.3:47839)
0620 23:39:10.508448 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:10.508557 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":281,"threads_started":1}
I20260620 23:39:10.572566 15834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:10.571825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41617 (local address 127.0.97.1:40829)
0620 23:39:10.572319 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:10.572401 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":269,"threads_started":1}
I20260620 23:39:11.048842 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:11.048087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41617 (local address 127.0.97.2:40443)
0620 23:39:11.048576 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:11.048673 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":258,"threads_started":1}
I20260620 23:39:11.510879 15836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:11.509853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41617 (local address 127.0.97.3:38513)
0620 23:39:11.510481 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:11.510661 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":303,"threads_started":1}
I20260620 23:39:11.574338 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:11.573490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41617 (local address 127.0.97.1:33229)
0620 23:39:11.574038 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:11.574128 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":274,"threads_started":1}
I20260620 23:39:12.050659 15838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:12.049828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41617 (local address 127.0.97.2:48279)
0620 23:39:12.050359 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:12.050448 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":213,"threads_started":1}
I20260620 23:39:12.512809 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:12.511910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41617 (local address 127.0.97.3:41697)
0620 23:39:12.512486 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:12.512578 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":217,"threads_started":1}
I20260620 23:39:12.576436 15840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:12.575617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41617 (local address 127.0.97.1:42825)
0620 23:39:12.576153 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:12.576261 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":309,"threads_started":1}
I20260620 23:39:13.052532 15841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:13.051776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41617 (local address 127.0.97.2:53117)
0620 23:39:13.052281 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:13.052370 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":252,"threads_started":1}
I20260620 23:39:13.515086 15842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:13.514026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41617 (local address 127.0.97.3:51837)
0620 23:39:13.514706 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:13.514833 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":339,"threads_started":1}
I20260620 23:39:13.578114 15843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:13.577339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41617 (local address 127.0.97.1:37409)
0620 23:39:13.577824 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:13.577933 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":247,"threads_started":1}
I20260620 23:39:14.054311 15844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:14.053515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41617 (local address 127.0.97.2:37241)
0620 23:39:14.054043 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:14.054132 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":243,"threads_started":1}
I20260620 23:39:14.516855 15845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:14.516071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41617 (local address 127.0.97.3:42377)
0620 23:39:14.516577 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:14.516667 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":235,"threads_started":1}
I20260620 23:39:14.579829 15846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:14.579146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41617 (local address 127.0.97.1:42725)
0620 23:39:14.579581 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:14.579660 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":229,"threads_started":1}
I20260620 23:39:15.056340 15847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:15.055566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41617 (local address 127.0.97.2:50689)
0620 23:39:15.056084 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:15.056174 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":276,"threads_started":1}
I20260620 23:39:15.518617 15848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:15.517872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41617 (local address 127.0.97.3:36209)
0620 23:39:15.518350 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:15.518440 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":255,"threads_started":1}
I20260620 23:39:15.581614 15849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:15.580822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41617 (local address 127.0.97.1:45643)
0620 23:39:15.581334 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:15.581426 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":299,"threads_started":1}
I20260620 23:39:16.058233 15850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:16.057444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41617 (local address 127.0.97.2:52231)
0620 23:39:16.057967 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:16.058055 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":290,"threads_started":1}
I20260620 23:39:16.520418 15851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:16.519699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41617 (local address 127.0.97.3:53003)
0620 23:39:16.520134 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:16.520225 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":218,"threads_started":1}
I20260620 23:39:16.583623 15852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:16.582784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41617 (local address 127.0.97.1:47649)
0620 23:39:16.583345 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:16.583442 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":331,"threads_started":1}
I20260620 23:39:17.060002 15853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:17.059295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41617 (local address 127.0.97.2:43457)
0620 23:39:17.059718 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:17.059798 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":212,"threads_started":1}
I20260620 23:39:17.522008 15854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:17.521349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41617 (local address 127.0.97.3:57503)
0620 23:39:17.521761 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:17.521847 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":211,"threads_started":1}
I20260620 23:39:17.585500 15855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:17.584635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41617 (local address 127.0.97.1:56483)
0620 23:39:17.585166 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:17.585280 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":259,"threads_started":1}
I20260620 23:39:18.061944 15856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:18.060980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41617 (local address 127.0.97.2:36175)
0620 23:39:18.061564 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:18.061662 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":283,"threads_started":1}
I20260620 23:39:18.523826 15857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:18.523066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41617 (local address 127.0.97.3:42345)
0620 23:39:18.523545 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:18.523636 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":224,"threads_started":1}
I20260620 23:39:18.587258 15858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:18.586471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41617 (local address 127.0.97.1:34819)
0620 23:39:18.586918 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:18.587071 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":213,"threads_started":1}
I20260620 23:39:19.063853 15859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:19.063066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41617 (local address 127.0.97.2:57603)
0620 23:39:19.063562 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:19.063651 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":246,"threads_started":1}
I20260620 23:39:19.525795 15860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:19.524934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41617 (local address 127.0.97.3:53291)
0620 23:39:19.525502 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:19.525607 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":335,"threads_started":1}
I20260620 23:39:19.589210 15861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:19.588369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41617 (local address 127.0.97.1:46347)
0620 23:39:19.588924 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:19.589011 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":320,"threads_started":1}
I20260620 23:39:20.065814 15862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:20.065046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41617 (local address 127.0.97.2:39953)
0620 23:39:20.065526 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:20.065621 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":240,"threads_started":1}
I20260620 23:39:20.507326 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 15390
I20260620 23:39:20.527769 15864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:20.527026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41617 (local address 127.0.97.3:37697)
0620 23:39:20.527499 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:20.527586 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":263,"threads_started":1}
I20260620 23:39:20.533371 15863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:20.508646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50076 (local address 127.0.97.1:45841)
0620 23:39:20.509633 (+ 987us) server_negotiation.cc:206] Beginning negotiation
0620 23:39:20.509653 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 23:39:20.509708 (+ 55us) server_negotiation.cc:407] Connection header received
0620 23:39:20.511831 (+ 2123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:39:20.511842 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:39:20.512329 (+ 487us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:39:20.512733 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:39:20.514351 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:20.515788 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:20.517188 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:20.517966 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:20.518182 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:20.526203 (+ 8021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:39:20.526213 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:39:20.526231 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:39:20.526283 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:39:20.529601 (+ 3318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:39:20.530533 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:39:20.530545 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:39:20.530558 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:39:20.530739 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:39:20.531282 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:39:20.531287 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:39:20.531289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:39:20.531867 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:39:20.532177 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0620 23:39:20.532657 (+ 480us) server_negotiation.cc:299] Negotiation successful
0620 23:39:20.533088 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":767,"thread_start_us":648,"threads_started":1}
I20260620 23:39:20.534782 15548 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:39:20.622757 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 15390
I20260620 23:39:20.640758 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 15556
I20260620 23:39:20.659350 15867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:20.641723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43726 (local address 127.0.97.2:39353)
0620 23:39:20.642315 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0620 23:39:20.642330 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 23:39:20.642368 (+ 38us) server_negotiation.cc:407] Connection header received
0620 23:39:20.644064 (+ 1696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:39:20.644075 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:39:20.644522 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:39:20.644824 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:39:20.646148 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:20.647526 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:20.648737 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:20.649417 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:20.649550 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:20.652341 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:39:20.652350 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:39:20.652366 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:39:20.652411 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:39:20.655764 (+ 3353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:39:20.656559 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:39:20.656566 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:39:20.656573 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:39:20.656756 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:39:20.657132 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:39:20.657136 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:39:20.657138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:39:20.657755 (+ 617us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:39:20.657998 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0620 23:39:20.658481 (+ 483us) server_negotiation.cc:299] Negotiation successful
0620 23:39:20.659055 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":289,"threads_started":1}
I20260620 23:39:20.660799 15679 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:39:20.748227 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 15556
I20260620 23:39:20.765349 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 15687
I20260620 23:39:20.784241 15869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:20.766403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32850 (local address 127.0.97.3:41913)
0620 23:39:20.767091 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0620 23:39:20.767105 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 23:39:20.767154 (+ 49us) server_negotiation.cc:407] Connection header received
0620 23:39:20.768817 (+ 1663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:39:20.768826 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:39:20.769278 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:39:20.769648 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:39:20.771330 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:20.772571 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:20.773895 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:20.774608 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:20.774813 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:20.777919 (+ 3106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:39:20.777928 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:39:20.777945 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:39:20.777993 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:39:20.781014 (+ 3021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:39:20.781687 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:39:20.781694 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:39:20.781701 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:39:20.781868 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:39:20.782206 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:39:20.782210 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:39:20.782212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:39:20.782741 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:39:20.783045 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0620 23:39:20.783477 (+ 432us) server_negotiation.cc:299] Negotiation successful
0620 23:39:20.783916 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":307,"threads_started":1}
I20260620 23:39:20.785723 15810 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:39:20.872287 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 15687
2026-06-20T23:39:20Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24715 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:39:20 dist-test-slave-tcgx krb5kdc[15873](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:39:20 dist-test-slave-tcgx krb5kdc[15873](info): set up 2 sockets
Jun 20 23:39:20 dist-test-slave-tcgx krb5kdc[15873](info): commencing operation
krb5kdc: starting...
W20260620 23:39:21.999915 388 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/krb5kdc/test-user.keytab.
Jun 20 23:39:22 dist-test-slave-tcgx krb5kdc[15873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:39:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:39:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 23:39:22.248775 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37589
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37589
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:39:22.586263 15886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:39:22.586584 15886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:39:22.586757 15886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:39:22.594291 15886 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 23:39:22.594408 15886 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 23:39:22.594478 15886 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 23:39:22.594558 15886 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:39:22.594651 15886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:39:22.594784 15886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:39:22.594875 15886 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:39:22.594985 15886 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:39:22.605326 15886 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37589
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:39:22.606711 15886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:39:22.608136 15886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:39:22.620517 15894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:39:22.620594 15892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:39:22.620857 15891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:39:22.621392 15886 server_base.cc:1034] running on GCE node
I20260620 23:39:22.622241 15886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:39:22.623847 15886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:39:22.625120 15886 hybrid_clock.cc:648] HybridClock initialized: now 1781998762625060 us; error 40 us; skew 500 ppm
I20260620 23:39:22.625483 15886 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:39:22 dist-test-slave-tcgx krb5kdc[15873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:39:22.628774 15886 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 23:39:22.630439 15886 webserver.cc:460] Webserver started at http://127.0.97.62:35631/ using document root <none> and password file <none>
I20260620 23:39:22.631286 15886 fs_manager.cc:359] Metadata directory not provided
I20260620 23:39:22.631425 15886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:39:22.631753 15886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:39:22.634836 15886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/master-0/data/instance:
uuid: "b891c2846180438488602be42d467117"
format_stamp: "Formatted at 2026-06-20 23:39:22 on dist-test-slave-tcgx"
server_key: "e941f9af04805e6454aa55d5cd65b181"
server_key_iv: "214e658b56ac36088703a664b5f2dbd8"
server_key_version: "encryptionkey@0"
I20260620 23:39:22.635735 15886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/master-0/wal/instance:
uuid: "b891c2846180438488602be42d467117"
format_stamp: "Formatted at 2026-06-20 23:39:22 on dist-test-slave-tcgx"
server_key: "e941f9af04805e6454aa55d5cd65b181"
server_key_iv: "214e658b56ac36088703a664b5f2dbd8"
server_key_version: "encryptionkey@0"
I20260620 23:39:22.641292 15886 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260620 23:39:22.645118 15901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:39:22.646678 15886 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 23:39:22.646893 15886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/master-0/wal
uuid: "b891c2846180438488602be42d467117"
format_stamp: "Formatted at 2026-06-20 23:39:22 on dist-test-slave-tcgx"
server_key: "e941f9af04805e6454aa55d5cd65b181"
server_key_iv: "214e658b56ac36088703a664b5f2dbd8"
server_key_version: "encryptionkey@0"
I20260620 23:39:22.647225 15886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:39:22.668164 15886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:39:22.672680 15886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:39:22.672973 15886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:39:22.692412 15886 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37589
I20260620 23:39:22.692423 15952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37589 every 8 connection(s)
I20260620 23:39:22.693921 15886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:39:22.698426 15953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:39:22.703466 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 15886
I20260620 23:39:22.703727 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/master-0/wal/instance
I20260620 23:39:22.704649 388 external_mini_cluster.cc:1428] Setting key c36bd3852eaa744e7e807fffe74f9bab
I20260620 23:39:22.710795 15953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b891c2846180438488602be42d467117: Bootstrap starting.
I20260620 23:39:22.714150 15953 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b891c2846180438488602be42d467117: Neither blocks nor log segments found. Creating new log.
I20260620 23:39:22.715323 15953 log.cc:824] T 00000000000000000000000000000000 P b891c2846180438488602be42d467117: Log is configured to *not* fsync() on all Append() calls
I20260620 23:39:22.718434 15953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b891c2846180438488602be42d467117: No bootstrap required, opened a new log
Jun 20 23:39:22 dist-test-slave-tcgx krb5kdc[15873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:39:22.728446 15953 raft_consensus.cc:348] T 00000000000000000000000000000000 P b891c2846180438488602be42d467117 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b891c2846180438488602be42d467117" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37589 } }
I20260620 23:39:22.728861 15953 raft_consensus.cc:374] T 00000000000000000000000000000000 P b891c2846180438488602be42d467117 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:39:22.729025 15953 raft_consensus.cc:729] T 00000000000000000000000000000000 P b891c2846180438488602be42d467117 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b891c2846180438488602be42d467117, State: Initialized, Role: FOLLOWER
I20260620 23:39:22.729681 15953 consensus_queue.cc:260] T 00000000000000000000000000000000 P b891c2846180438488602be42d467117 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b891c2846180438488602be42d467117" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37589 } }
I20260620 23:39:22.729910 15956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:22.707046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34294 (local address 127.0.97.62:37589)
0620 23:39:22.707913 (+ 867us) server_negotiation.cc:206] Beginning negotiation
0620 23:39:22.707925 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:39:22.707972 (+ 47us) server_negotiation.cc:407] Connection header received
0620 23:39:22.709774 (+ 1802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:39:22.709798 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:39:22.710208 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:39:22.710644 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:39:22.712109 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:22.713480 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:22.715069 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:22.716192 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:22.716338 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:22.721081 (+ 4743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:39:22.721090 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:39:22.721106 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:39:22.721146 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:39:22.724852 (+ 3706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:39:22.725798 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:39:22.725807 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:39:22.725814 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:39:22.726046 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:39:22.726530 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:39:22.726533 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:39:22.726535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:39:22.727263 (+ 728us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:39:22.727514 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0620 23:39:22.727961 (+ 447us) server_negotiation.cc:299] Negotiation successful
0620 23:39:22.728393 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":423,"threads_started":1}
I20260620 23:39:22.729940 15953 raft_consensus.cc:388] T 00000000000000000000000000000000 P b891c2846180438488602be42d467117 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:39:22.730141 15953 raft_consensus.cc:482] T 00000000000000000000000000000000 P b891c2846180438488602be42d467117 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:39:22.730357 15953 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b891c2846180438488602be42d467117 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:39:22.733274 15953 raft_consensus.cc:504] T 00000000000000000000000000000000 P b891c2846180438488602be42d467117 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b891c2846180438488602be42d467117" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37589 } }
I20260620 23:39:22.733760 15953 leader_election.cc:302] T 00000000000000000000000000000000 P b891c2846180438488602be42d467117 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b891c2846180438488602be42d467117; no voters:
I20260620 23:39:22.734329 15953 leader_election.cc:288] T 00000000000000000000000000000000 P b891c2846180438488602be42d467117 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:39:22.734480 15958 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b891c2846180438488602be42d467117 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:39:22.735237 15958 raft_consensus.cc:686] T 00000000000000000000000000000000 P b891c2846180438488602be42d467117 [term 1 LEADER]: Becoming Leader. State: Replica: b891c2846180438488602be42d467117, State: Running, Role: LEADER
I20260620 23:39:22.735859 15958 consensus_queue.cc:237] T 00000000000000000000000000000000 P b891c2846180438488602be42d467117 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b891c2846180438488602be42d467117" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37589 } }
I20260620 23:39:22.736501 15953 sys_catalog.cc:564] T 00000000000000000000000000000000 P b891c2846180438488602be42d467117 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:39:22.738771 15960 sys_catalog.cc:455] T 00000000000000000000000000000000 P b891c2846180438488602be42d467117 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b891c2846180438488602be42d467117. Latest consensus state: current_term: 1 leader_uuid: "b891c2846180438488602be42d467117" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b891c2846180438488602be42d467117" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37589 } } }
I20260620 23:39:22.738723 15959 sys_catalog.cc:455] T 00000000000000000000000000000000 P b891c2846180438488602be42d467117 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b891c2846180438488602be42d467117" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b891c2846180438488602be42d467117" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37589 } } }
I20260620 23:39:22.739207 15960 sys_catalog.cc:458] T 00000000000000000000000000000000 P b891c2846180438488602be42d467117 [sys.catalog]: This master's current role is: LEADER
I20260620 23:39:22.739217 15959 sys_catalog.cc:458] T 00000000000000000000000000000000 P b891c2846180438488602be42d467117 [sys.catalog]: This master's current role is: LEADER
I20260620 23:39:22.742381 15966 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:39:22.748606 15966 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:39:22.758759 15966 catalog_manager.cc:1347] Generated new cluster ID: 6bfd6b9c73a3459584b5ce669768df17
I20260620 23:39:22.758924 15966 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:39:22.780650 15966 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:39:22.782151 15966 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:39:22.794231 15966 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b891c2846180438488602be42d467117: Generated new TSK 0
I20260620 23:39:22.795200 15966 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 23:39:22.861099 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37589
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:39:23.234694 15979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:39:23.235085 15979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:39:23.235204 15979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:39:23.243245 15979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:39:23.243366 15979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:39:23.243480 15979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 23:39:23.253913 15979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37589
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:39:23.255323 15979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:39:23.256636 15979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:39:23.270454 15984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:39:23.270516 15985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:39:23.270354 15987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:39:23.271275 15979 server_base.cc:1034] running on GCE node
I20260620 23:39:23.271950 15979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:39:23.272946 15979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:39:23.274209 15979 hybrid_clock.cc:648] HybridClock initialized: now 1781998763274128 us; error 65 us; skew 500 ppm
I20260620 23:39:23.274560 15979 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:39:23 dist-test-slave-tcgx krb5kdc[15873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:39:23.277752 15979 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 23:39:23.279264 15979 webserver.cc:460] Webserver started at http://127.0.97.1:42213/ using document root <none> and password file <none>
I20260620 23:39:23.279976 15979 fs_manager.cc:359] Metadata directory not provided
I20260620 23:39:23.280128 15979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:39:23.280444 15979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:39:23.283532 15979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-0/data/instance:
uuid: "dacd9f26586346c0ab4b9922f88308b3"
format_stamp: "Formatted at 2026-06-20 23:39:23 on dist-test-slave-tcgx"
server_key: "4ddd36bdf282d4bd73ad7efdc123b3f6"
server_key_iv: "a80e365783dc04e479250ba41387a93c"
server_key_version: "encryptionkey@0"
I20260620 23:39:23.284305 15979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-0/wal/instance:
uuid: "dacd9f26586346c0ab4b9922f88308b3"
format_stamp: "Formatted at 2026-06-20 23:39:23 on dist-test-slave-tcgx"
server_key: "4ddd36bdf282d4bd73ad7efdc123b3f6"
server_key_iv: "a80e365783dc04e479250ba41387a93c"
server_key_version: "encryptionkey@0"
I20260620 23:39:23.289587 15979 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:39:23.293174 15994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:39:23.294647 15979 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:39:23.294835 15979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-0/wal
uuid: "dacd9f26586346c0ab4b9922f88308b3"
format_stamp: "Formatted at 2026-06-20 23:39:23 on dist-test-slave-tcgx"
server_key: "4ddd36bdf282d4bd73ad7efdc123b3f6"
server_key_iv: "a80e365783dc04e479250ba41387a93c"
server_key_version: "encryptionkey@0"
I20260620 23:39:23.295073 15979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:39:23.313694 15979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:39:23.318104 15979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:39:23.318429 15979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:39:23.319558 15979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:39:23.320997 15979 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:39:23.321133 15979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:39:23.321295 15979 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:39:23.321369 15979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:39:23.355455 15979 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45245
I20260620 23:39:23.355504 16106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45245 every 8 connection(s)
I20260620 23:39:23.356832 15979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
I20260620 23:39:23.359647 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 15979
I20260620 23:39:23.359885 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-0/wal/instance
I20260620 23:39:23.360575 388 external_mini_cluster.cc:1428] Setting key 67f71c97d8a8fe97598754d7eb0999dc
Jun 20 23:39:23 dist-test-slave-tcgx krb5kdc[15873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:39:23.380381 16109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:23.359631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56295 (local address 127.0.97.62:37589)
0620 23:39:23.360151 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0620 23:39:23.360157 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:39:23.361373 (+ 1216us) server_negotiation.cc:407] Connection header received
0620 23:39:23.363823 (+ 2450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:39:23.363828 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:39:23.363897 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:39:23.363990 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:39:23.366290 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:23.367208 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:23.369877 (+ 2669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:23.370293 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:23.370407 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:23.374356 (+ 3949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:39:23.374361 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:39:23.374365 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:39:23.374401 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:39:23.377106 (+ 2705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:39:23.377904 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:39:23.377912 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:39:23.377916 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:39:23.378049 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:39:23.378500 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:39:23.378507 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:39:23.378510 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:39:23.378797 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:39:23.378984 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 23:39:23.379745 (+ 761us) server_negotiation.cc:299] Negotiation successful
0620 23:39:23.379991 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":262,"threads_started":1}
I20260620 23:39:23.381565 16110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:23.360255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37589 (local address 127.0.97.1:56295)
0620 23:39:23.361165 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:23.361230 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 23:39:23.363510 (+ 2280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:39:23.364299 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:39:23.364314 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:39:23.365185 (+ 871us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:39:23.366048 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:39:23.366087 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:39:23.367388 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:39:23.367396 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:39:23.369669 (+ 2273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:39:23.369690 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:39:23.370555 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:39:23.370562 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:39:23.370909 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:23.371793 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:39:23.371816 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:39:23.374094 (+ 2278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:39:23.377275 (+ 3181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:39:23.377285 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:39:23.377305 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:39:23.377773 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:39:23.378170 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:39:23.378175 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:39:23.378179 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:39:23.378387 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:39:23.378953 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:39:23.378963 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:39:23.379352 (+ 389us) client_negotiation.cc:769] Sending connection context
0620 23:39:23.379646 (+ 294us) client_negotiation.cc:240] Negotiation successful
0620 23:39:23.380245 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":370,"threads_started":1}
I20260620 23:39:23.383726 16107 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37589
I20260620 23:39:23.384088 16107 heartbeater.cc:464] Registering TS with master...
I20260620 23:39:23.384883 16107 heartbeater.cc:511] Master 127.0.97.62:37589 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 23:39:23.387418 15918 ts_manager.cc:194] Registered new tserver with Master: dacd9f26586346c0ab4b9922f88308b3 (127.0.97.1:45245)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260620 23:39:23.389128 15918 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:56295
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 23:39:23.420038 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37589
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:39:23.771708 16113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:39:23.772015 16113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:39:23.772181 16113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:39:23.779536 16113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:39:23.779655 16113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:39:23.779841 16113 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 23:39:23.790313 16113 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37589
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:39:23.791846 16113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:39:23.793210 16113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:39:23.806849 16118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:39:23.806900 16121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:39:23.807124 16119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:39:23.807938 16113 server_base.cc:1034] running on GCE node
I20260620 23:39:23.808629 16113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:39:23.809681 16113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:39:23.810897 16113 hybrid_clock.cc:648] HybridClock initialized: now 1781998763810854 us; error 37 us; skew 500 ppm
I20260620 23:39:23.811338 16113 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:39:23 dist-test-slave-tcgx krb5kdc[15873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:39:23.814549 16113 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 23:39:23.816005 16113 webserver.cc:460] Webserver started at http://127.0.97.2:33373/ using document root <none> and password file <none>
I20260620 23:39:23.816869 16113 fs_manager.cc:359] Metadata directory not provided
I20260620 23:39:23.816973 16113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:39:23.817250 16113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:39:23.820236 16113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-1/data/instance:
uuid: "484d1f530fc44784a3be857749e2aec1"
format_stamp: "Formatted at 2026-06-20 23:39:23 on dist-test-slave-tcgx"
server_key: "3f903f91fa429b32b3990ca10bc07ecc"
server_key_iv: "78277e1329cf21e3a17e33fad220cc3e"
server_key_version: "encryptionkey@0"
I20260620 23:39:23.820942 16113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-1/wal/instance:
uuid: "484d1f530fc44784a3be857749e2aec1"
format_stamp: "Formatted at 2026-06-20 23:39:23 on dist-test-slave-tcgx"
server_key: "3f903f91fa429b32b3990ca10bc07ecc"
server_key_iv: "78277e1329cf21e3a17e33fad220cc3e"
server_key_version: "encryptionkey@0"
I20260620 23:39:23.826202 16113 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 23:39:23.829697 16128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:39:23.831131 16113 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:39:23.831293 16113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-1/wal
uuid: "484d1f530fc44784a3be857749e2aec1"
format_stamp: "Formatted at 2026-06-20 23:39:23 on dist-test-slave-tcgx"
server_key: "3f903f91fa429b32b3990ca10bc07ecc"
server_key_iv: "78277e1329cf21e3a17e33fad220cc3e"
server_key_version: "encryptionkey@0"
I20260620 23:39:23.831516 16113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:39:23.893074 16113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:39:23.897600 16113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:39:23.897912 16113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:39:23.899060 16113 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:39:23.900537 16113 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:39:23.900612 16113 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:39:23.900760 16113 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:39:23.900818 16113 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:39:23.935492 16113 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46215
I20260620 23:39:23.935555 16240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46215 every 8 connection(s)
I20260620 23:39:23.936908 16113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
I20260620 23:39:23.945184 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 16113
I20260620 23:39:23.945376 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-1/wal/instance
I20260620 23:39:23.945829 388 external_mini_cluster.cc:1428] Setting key 15ba15bbd068b11899b3268b21ea54e6
Jun 20 23:39:23 dist-test-slave-tcgx krb5kdc[15873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:39:23.961064 16243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:23.939645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55557 (local address 127.0.97.62:37589)
0620 23:39:23.940145 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0620 23:39:23.940150 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:39:23.941050 (+ 900us) server_negotiation.cc:407] Connection header received
0620 23:39:23.943665 (+ 2615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:39:23.943671 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:39:23.943776 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:39:23.943902 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:39:23.945650 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:23.946482 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:23.948634 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:23.949126 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:23.949226 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:23.953875 (+ 4649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:39:23.953884 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:39:23.953892 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:39:23.953943 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:39:23.957288 (+ 3345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:39:23.958410 (+ 1122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:39:23.958414 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:39:23.958417 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:39:23.958560 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:39:23.959129 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:39:23.959133 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:39:23.959136 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:39:23.959409 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:39:23.959542 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 23:39:23.960589 (+ 1047us) server_negotiation.cc:299] Negotiation successful
0620 23:39:23.960781 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":250,"threads_started":1}
I20260620 23:39:23.962706 16244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:23.940129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37589 (local address 127.0.97.2:55557)
0620 23:39:23.940855 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:23.940893 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 23:39:23.943348 (+ 2455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:39:23.944169 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:39:23.944178 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:39:23.944753 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:39:23.945459 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:39:23.945486 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:39:23.946686 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:39:23.946692 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:39:23.948446 (+ 1754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:39:23.948463 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:39:23.949384 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:39:23.949393 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:39:23.949778 (+ 385us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:23.950555 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:39:23.950591 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:39:23.953587 (+ 2996us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:39:23.957485 (+ 3898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:39:23.957497 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:39:23.957517 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:39:23.958263 (+ 746us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:39:23.958680 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:39:23.958688 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:39:23.958694 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:39:23.958999 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:39:23.959542 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:39:23.959554 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:39:23.960097 (+ 543us) client_negotiation.cc:769] Sending connection context
0620 23:39:23.960455 (+ 358us) client_negotiation.cc:240] Negotiation successful
0620 23:39:23.961109 (+ 654us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":287,"threads_started":1}
I20260620 23:39:23.964354 16241 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37589
I20260620 23:39:23.964696 16241 heartbeater.cc:464] Registering TS with master...
I20260620 23:39:23.965641 16241 heartbeater.cc:511] Master 127.0.97.62:37589 requested a full tablet report, sending...
I20260620 23:39:23.967459 15918 ts_manager.cc:194] Registered new tserver with Master: 484d1f530fc44784a3be857749e2aec1 (127.0.97.2:46215)
I20260620 23:39:23.968293 15918 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:55557
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 23:39:24.004401 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37589
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:39:24.344175 16247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:39:24.344525 16247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:39:24.344705 16247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:39:24.352304 16247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:39:24.352470 16247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:39:24.352648 16247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 23:39:24.362843 16247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37589
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:39:24.364297 16247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:39:24.365727 16247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:39:24.379900 16253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:39:24.379879 16252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:39:24.379854 16255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:39:24.380895 16247 server_base.cc:1034] running on GCE node
I20260620 23:39:24.381592 16247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:39:24.382612 16247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:39:24.383843 16247 hybrid_clock.cc:648] HybridClock initialized: now 1781998764383724 us; error 109 us; skew 500 ppm
I20260620 23:39:24.384258 16247 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:39:24 dist-test-slave-tcgx krb5kdc[15873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998764, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:39:24.387832 16247 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 23:39:24.389426 16247 webserver.cc:460] Webserver started at http://127.0.97.3:35389/ using document root <none> and password file <none>
I20260620 23:39:24.390184 16247 fs_manager.cc:359] Metadata directory not provided
I20260620 23:39:24.390334 16247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:39:24.390678 16247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:39:24.393241 16107 heartbeater.cc:503] Master 127.0.97.62:37589 was elected leader, sending a full tablet report...
I20260620 23:39:24.394073 16247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-2/data/instance:
uuid: "e85b9450f44f4ba097cca79c9d37669c"
format_stamp: "Formatted at 2026-06-20 23:39:24 on dist-test-slave-tcgx"
server_key: "e1740287c5d986a49ac6fcf0d0cf1805"
server_key_iv: "fc3d60b0e3ed9d7ffe927cdac0327937"
server_key_version: "encryptionkey@0"
I20260620 23:39:24.395164 16247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-2/wal/instance:
uuid: "e85b9450f44f4ba097cca79c9d37669c"
format_stamp: "Formatted at 2026-06-20 23:39:24 on dist-test-slave-tcgx"
server_key: "e1740287c5d986a49ac6fcf0d0cf1805"
server_key_iv: "fc3d60b0e3ed9d7ffe927cdac0327937"
server_key_version: "encryptionkey@0"
I20260620 23:39:24.400825 16247 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 23:39:24.405223 16262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:39:24.407542 16247 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260620 23:39:24.407750 16247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-2/wal
uuid: "e85b9450f44f4ba097cca79c9d37669c"
format_stamp: "Formatted at 2026-06-20 23:39:24 on dist-test-slave-tcgx"
server_key: "e1740287c5d986a49ac6fcf0d0cf1805"
server_key_iv: "fc3d60b0e3ed9d7ffe927cdac0327937"
server_key_version: "encryptionkey@0"
I20260620 23:39:24.407999 16247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:39:24.423357 16247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:39:24.428377 16247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:39:24.428710 16247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:39:24.429793 16247 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:39:24.431404 16247 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:39:24.431493 16247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:39:24.431656 16247 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:39:24.431712 16247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:39:24.466147 16247 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42459
I20260620 23:39:24.466184 16374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42459 every 8 connection(s)
I20260620 23:39:24.467645 16247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
I20260620 23:39:24.469458 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 16247
I20260620 23:39:24.469669 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-2/wal/instance
I20260620 23:39:24.470317 388 external_mini_cluster.cc:1428] Setting key cb5e28adeff3ac8eb0ecd6dafae5322f
Jun 20 23:39:24 dist-test-slave-tcgx krb5kdc[15873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998764, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:39:24.489395 16377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:24.470667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33611 (local address 127.0.97.62:37589)
0620 23:39:24.471239 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0620 23:39:24.471245 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:39:24.472567 (+ 1322us) server_negotiation.cc:407] Connection header received
0620 23:39:24.474691 (+ 2124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:39:24.474696 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:39:24.474781 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:39:24.474882 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:39:24.476514 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:24.477396 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:24.478905 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:24.479452 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:24.479561 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:24.483042 (+ 3481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:39:24.483047 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:39:24.483050 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:39:24.483098 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:39:24.486501 (+ 3403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:39:24.487326 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:39:24.487329 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:39:24.487332 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:39:24.487452 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:39:24.487844 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:39:24.487847 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:39:24.487849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:39:24.488125 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:39:24.488234 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 23:39:24.488926 (+ 692us) server_negotiation.cc:299] Negotiation successful
0620 23:39:24.489098 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":322,"threads_started":1}
I20260620 23:39:24.490383 16378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:24.471395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37589 (local address 127.0.97.3:33611)
0620 23:39:24.472362 (+ 967us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:24.472413 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 23:39:24.474435 (+ 2022us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:39:24.475199 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:39:24.475208 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:39:24.475743 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:39:24.476346 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:39:24.476363 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:39:24.477533 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:39:24.477536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:39:24.478773 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:39:24.478784 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:39:24.479655 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:39:24.479660 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:39:24.479868 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:24.480559 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:39:24.480581 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:39:24.482768 (+ 2187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:39:24.486638 (+ 3870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:39:24.486646 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:39:24.486660 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:39:24.487212 (+ 552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:39:24.487547 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:39:24.487551 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:39:24.487553 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:39:24.487739 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:39:24.488229 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:39:24.488235 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:39:24.488567 (+ 332us) client_negotiation.cc:769] Sending connection context
0620 23:39:24.488795 (+ 228us) client_negotiation.cc:240] Negotiation successful
0620 23:39:24.489231 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":361,"threads_started":1}
I20260620 23:39:24.491814 16375 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37589
I20260620 23:39:24.492143 16375 heartbeater.cc:464] Registering TS with master...
I20260620 23:39:24.492895 16375 heartbeater.cc:511] Master 127.0.97.62:37589 requested a full tablet report, sending...
I20260620 23:39:24.494360 15918 ts_manager.cc:194] Registered new tserver with Master: e85b9450f44f4ba097cca79c9d37669c (127.0.97.3:42459)
I20260620 23:39:24.495270 15918 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:33611
I20260620 23:39:24.498860 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:39:24.517220 16377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:24.503781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34308 (local address 127.0.97.62:37589)
0620 23:39:24.503983 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 23:39:24.503992 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:39:24.504446 (+ 454us) server_negotiation.cc:407] Connection header received
0620 23:39:24.504655 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:39:24.504661 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:39:24.504767 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:39:24.504883 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:39:24.506385 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:24.507206 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:24.508572 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:24.508986 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:24.509136 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:24.511699 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:39:24.511704 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:39:24.511708 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:39:24.511744 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:39:24.514363 (+ 2619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:39:24.515164 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:39:24.515168 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:39:24.515170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:39:24.515293 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:39:24.515690 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:39:24.515695 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:39:24.515701 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:39:24.516037 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:39:24.516327 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0620 23:39:24.516621 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 23:39:24.516868 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 23:39:24.521541 15918 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34308:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:39:24.528625 15918 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:39:24.552615 16389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:24.541821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54176 (local address 127.0.97.2:46215)
0620 23:39:24.543983 (+ 2162us) server_negotiation.cc:206] Beginning negotiation
0620 23:39:24.543994 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:39:24.544024 (+ 30us) server_negotiation.cc:407] Connection header received
0620 23:39:24.544140 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:39:24.544148 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:39:24.544431 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:39:24.544651 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:39:24.545503 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:24.546906 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:24.549472 (+ 2566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:24.551189 (+ 1717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:24.551335 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:24.551577 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0620 23:39:24.552042 (+ 465us) server_negotiation.cc:299] Negotiation successful
0620 23:39:24.552228 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1984,"thread_start_us":1870,"threads_started":1}
I20260620 23:39:24.552587 16388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:24.541582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46215 (local address 127.0.0.1:54176)
0620 23:39:24.543135 (+ 1553us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:24.543184 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 23:39:24.543388 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:39:24.544713 (+ 1325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:39:24.544720 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:39:24.544752 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:39:24.545282 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:39:24.545301 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:39:24.547132 (+ 1831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:39:24.547138 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:39:24.549213 (+ 2075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:39:24.549237 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:39:24.551398 (+ 2161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:39:24.551406 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:39:24.551757 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:24.551818 (+ 61us) client_negotiation.cc:769] Sending connection context
0620 23:39:24.551963 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 23:39:24.552210 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1182,"thread_start_us":1022,"threads_started":1}
I20260620 23:39:24.554690 16392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:24.544566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45245 (local address 127.0.0.1:34370)
0620 23:39:24.545401 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:24.545421 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:39:24.545538 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:39:24.546828 (+ 1290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:39:24.546836 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:39:24.546865 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:39:24.547364 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:39:24.547375 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:39:24.549334 (+ 1959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:39:24.549340 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:39:24.551313 (+ 1973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:39:24.551327 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:39:24.553679 (+ 2352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:39:24.553686 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:39:24.553876 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:24.553917 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 23:39:24.554046 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 23:39:24.554220 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"mutex_wait_us":36,"thread_start_us":821,"threads_started":1}
I20260620 23:39:24.555119 16390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:24.541817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42459 (local address 127.0.0.1:56564)
0620 23:39:24.544366 (+ 2549us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:24.544380 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:39:24.544536 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:39:24.546199 (+ 1663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:39:24.546205 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:39:24.546234 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:39:24.546652 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:39:24.546669 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:39:24.548835 (+ 2166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:39:24.548841 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:39:24.551471 (+ 2630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:39:24.551487 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:39:24.554086 (+ 2599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:39:24.554093 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:39:24.554271 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:24.554305 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 23:39:24.554406 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 23:39:24.554552 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2453,"mutex_wait_us":5,"thread_start_us":2346,"threads_started":1}
I20260620 23:39:24.555532 16391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:24.542471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56564 (local address 127.0.97.3:42459)
0620 23:39:24.544048 (+ 1577us) server_negotiation.cc:206] Beginning negotiation
0620 23:39:24.544058 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:39:24.545662 (+ 1604us) server_negotiation.cc:407] Connection header received
0620 23:39:24.545779 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:39:24.545786 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:39:24.546031 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:39:24.546211 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:39:24.546892 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:24.548648 (+ 1756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:24.551688 (+ 3040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:24.553531 (+ 1843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:24.553677 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:24.553915 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0620 23:39:24.554905 (+ 990us) server_negotiation.cc:299] Negotiation successful
0620 23:39:24.555138 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1395,"thread_start_us":1408,"threads_started":1}
I20260620 23:39:24.556607 16393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:24.545207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34370 (local address 127.0.97.1:45245)
0620 23:39:24.546217 (+ 1010us) server_negotiation.cc:206] Beginning negotiation
0620 23:39:24.546225 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:39:24.546251 (+ 26us) server_negotiation.cc:407] Connection header received
0620 23:39:24.546355 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:39:24.546365 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:39:24.546614 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:39:24.546828 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:39:24.547556 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:24.549194 (+ 1638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:24.551538 (+ 2344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:24.553412 (+ 1874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:24.553601 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:24.553852 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0620 23:39:24.554089 (+ 237us) server_negotiation.cc:299] Negotiation successful
0620 23:39:24.554251 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":834,"thread_start_us":651,"threads_started":1}
I20260620 23:39:24.557718 16176 tablet_service.cc:1467] Processing CreateTablet for tablet 0801f66980684fe8baafd65b49b0cd83 (DEFAULT_TABLE table=test-table [id=9ebdaeb759024a6d8ed6855ae1d35910]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:39:24.559317 16176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0801f66980684fe8baafd65b49b0cd83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:39:24.559381 16310 tablet_service.cc:1467] Processing CreateTablet for tablet 0801f66980684fe8baafd65b49b0cd83 (DEFAULT_TABLE table=test-table [id=9ebdaeb759024a6d8ed6855ae1d35910]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:39:24.560222 16042 tablet_service.cc:1467] Processing CreateTablet for tablet 0801f66980684fe8baafd65b49b0cd83 (DEFAULT_TABLE table=test-table [id=9ebdaeb759024a6d8ed6855ae1d35910]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:39:24.560935 16310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0801f66980684fe8baafd65b49b0cd83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:39:24.561481 16042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0801f66980684fe8baafd65b49b0cd83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:39:24.574681 16394 tablet_bootstrap.cc:492] T 0801f66980684fe8baafd65b49b0cd83 P dacd9f26586346c0ab4b9922f88308b3: Bootstrap starting.
I20260620 23:39:24.577154 16395 tablet_bootstrap.cc:492] T 0801f66980684fe8baafd65b49b0cd83 P 484d1f530fc44784a3be857749e2aec1: Bootstrap starting.
I20260620 23:39:24.579210 16394 tablet_bootstrap.cc:654] T 0801f66980684fe8baafd65b49b0cd83 P dacd9f26586346c0ab4b9922f88308b3: Neither blocks nor log segments found. Creating new log.
I20260620 23:39:24.579735 16396 tablet_bootstrap.cc:492] T 0801f66980684fe8baafd65b49b0cd83 P e85b9450f44f4ba097cca79c9d37669c: Bootstrap starting.
I20260620 23:39:24.581120 16395 tablet_bootstrap.cc:654] T 0801f66980684fe8baafd65b49b0cd83 P 484d1f530fc44784a3be857749e2aec1: Neither blocks nor log segments found. Creating new log.
I20260620 23:39:24.581333 16394 log.cc:824] T 0801f66980684fe8baafd65b49b0cd83 P dacd9f26586346c0ab4b9922f88308b3: Log is configured to *not* fsync() on all Append() calls
I20260620 23:39:24.582520 16395 log.cc:824] T 0801f66980684fe8baafd65b49b0cd83 P 484d1f530fc44784a3be857749e2aec1: Log is configured to *not* fsync() on all Append() calls
I20260620 23:39:24.583389 16396 tablet_bootstrap.cc:654] T 0801f66980684fe8baafd65b49b0cd83 P e85b9450f44f4ba097cca79c9d37669c: Neither blocks nor log segments found. Creating new log.
I20260620 23:39:24.584496 16396 log.cc:824] T 0801f66980684fe8baafd65b49b0cd83 P e85b9450f44f4ba097cca79c9d37669c: Log is configured to *not* fsync() on all Append() calls
I20260620 23:39:24.585094 16394 tablet_bootstrap.cc:492] T 0801f66980684fe8baafd65b49b0cd83 P dacd9f26586346c0ab4b9922f88308b3: No bootstrap required, opened a new log
I20260620 23:39:24.585415 16394 ts_tablet_manager.cc:1397] T 0801f66980684fe8baafd65b49b0cd83 P dacd9f26586346c0ab4b9922f88308b3: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260620 23:39:24.585872 16395 tablet_bootstrap.cc:492] T 0801f66980684fe8baafd65b49b0cd83 P 484d1f530fc44784a3be857749e2aec1: No bootstrap required, opened a new log
I20260620 23:39:24.586170 16395 ts_tablet_manager.cc:1397] T 0801f66980684fe8baafd65b49b0cd83 P 484d1f530fc44784a3be857749e2aec1: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260620 23:39:24.587297 16396 tablet_bootstrap.cc:492] T 0801f66980684fe8baafd65b49b0cd83 P e85b9450f44f4ba097cca79c9d37669c: No bootstrap required, opened a new log
I20260620 23:39:24.587545 16396 ts_tablet_manager.cc:1397] T 0801f66980684fe8baafd65b49b0cd83 P e85b9450f44f4ba097cca79c9d37669c: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 23:39:24.595216 16396 raft_consensus.cc:348] T 0801f66980684fe8baafd65b49b0cd83 P e85b9450f44f4ba097cca79c9d37669c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484d1f530fc44784a3be857749e2aec1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46215 } } peers { permanent_uuid: "e85b9450f44f4ba097cca79c9d37669c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42459 } } peers { permanent_uuid: "dacd9f26586346c0ab4b9922f88308b3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45245 } }
I20260620 23:39:24.595633 16396 raft_consensus.cc:374] T 0801f66980684fe8baafd65b49b0cd83 P e85b9450f44f4ba097cca79c9d37669c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:39:24.595736 16396 raft_consensus.cc:729] T 0801f66980684fe8baafd65b49b0cd83 P e85b9450f44f4ba097cca79c9d37669c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e85b9450f44f4ba097cca79c9d37669c, State: Initialized, Role: FOLLOWER
I20260620 23:39:24.596390 16396 consensus_queue.cc:260] T 0801f66980684fe8baafd65b49b0cd83 P e85b9450f44f4ba097cca79c9d37669c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484d1f530fc44784a3be857749e2aec1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46215 } } peers { permanent_uuid: "e85b9450f44f4ba097cca79c9d37669c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42459 } } peers { permanent_uuid: "dacd9f26586346c0ab4b9922f88308b3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45245 } }
I20260620 23:39:24.597519 16375 heartbeater.cc:503] Master 127.0.97.62:37589 was elected leader, sending a full tablet report...
I20260620 23:39:24.597699 16394 raft_consensus.cc:348] T 0801f66980684fe8baafd65b49b0cd83 P dacd9f26586346c0ab4b9922f88308b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484d1f530fc44784a3be857749e2aec1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46215 } } peers { permanent_uuid: "e85b9450f44f4ba097cca79c9d37669c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42459 } } peers { permanent_uuid: "dacd9f26586346c0ab4b9922f88308b3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45245 } }
I20260620 23:39:24.598172 16394 raft_consensus.cc:374] T 0801f66980684fe8baafd65b49b0cd83 P dacd9f26586346c0ab4b9922f88308b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:39:24.597920 16395 raft_consensus.cc:348] T 0801f66980684fe8baafd65b49b0cd83 P 484d1f530fc44784a3be857749e2aec1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484d1f530fc44784a3be857749e2aec1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46215 } } peers { permanent_uuid: "e85b9450f44f4ba097cca79c9d37669c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42459 } } peers { permanent_uuid: "dacd9f26586346c0ab4b9922f88308b3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45245 } }
I20260620 23:39:24.598278 16394 raft_consensus.cc:729] T 0801f66980684fe8baafd65b49b0cd83 P dacd9f26586346c0ab4b9922f88308b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dacd9f26586346c0ab4b9922f88308b3, State: Initialized, Role: FOLLOWER
I20260620 23:39:24.598344 16395 raft_consensus.cc:374] T 0801f66980684fe8baafd65b49b0cd83 P 484d1f530fc44784a3be857749e2aec1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:39:24.598448 16395 raft_consensus.cc:729] T 0801f66980684fe8baafd65b49b0cd83 P 484d1f530fc44784a3be857749e2aec1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 484d1f530fc44784a3be857749e2aec1, State: Initialized, Role: FOLLOWER
I20260620 23:39:24.599087 16394 consensus_queue.cc:260] T 0801f66980684fe8baafd65b49b0cd83 P dacd9f26586346c0ab4b9922f88308b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484d1f530fc44784a3be857749e2aec1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46215 } } peers { permanent_uuid: "e85b9450f44f4ba097cca79c9d37669c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42459 } } peers { permanent_uuid: "dacd9f26586346c0ab4b9922f88308b3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45245 } }
I20260620 23:39:24.599150 16395 consensus_queue.cc:260] T 0801f66980684fe8baafd65b49b0cd83 P 484d1f530fc44784a3be857749e2aec1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484d1f530fc44784a3be857749e2aec1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46215 } } peers { permanent_uuid: "e85b9450f44f4ba097cca79c9d37669c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42459 } } peers { permanent_uuid: "dacd9f26586346c0ab4b9922f88308b3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45245 } }
I20260620 23:39:24.599573 16396 ts_tablet_manager.cc:1428] T 0801f66980684fe8baafd65b49b0cd83 P e85b9450f44f4ba097cca79c9d37669c: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260620 23:39:24.600625 16241 heartbeater.cc:503] Master 127.0.97.62:37589 was elected leader, sending a full tablet report...
I20260620 23:39:24.602805 16395 ts_tablet_manager.cc:1428] T 0801f66980684fe8baafd65b49b0cd83 P 484d1f530fc44784a3be857749e2aec1: Time spent starting tablet: real 0.017s user 0.016s sys 0.001s
I20260620 23:39:24.603327 16394 ts_tablet_manager.cc:1428] T 0801f66980684fe8baafd65b49b0cd83 P dacd9f26586346c0ab4b9922f88308b3: Time spent starting tablet: real 0.018s user 0.010s sys 0.008s
W20260620 23:39:24.609261 16108 tablet.cc:2369] T 0801f66980684fe8baafd65b49b0cd83 P dacd9f26586346c0ab4b9922f88308b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:39:24.689030 16242 tablet.cc:2369] T 0801f66980684fe8baafd65b49b0cd83 P 484d1f530fc44784a3be857749e2aec1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:39:24.719506 16376 tablet.cc:2369] T 0801f66980684fe8baafd65b49b0cd83 P e85b9450f44f4ba097cca79c9d37669c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:39:24.802013 16402 raft_consensus.cc:482] T 0801f66980684fe8baafd65b49b0cd83 P 484d1f530fc44784a3be857749e2aec1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:39:24.802353 16402 raft_consensus.cc:504] T 0801f66980684fe8baafd65b49b0cd83 P 484d1f530fc44784a3be857749e2aec1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484d1f530fc44784a3be857749e2aec1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46215 } } peers { permanent_uuid: "e85b9450f44f4ba097cca79c9d37669c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42459 } } peers { permanent_uuid: "dacd9f26586346c0ab4b9922f88308b3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45245 } }
I20260620 23:39:24.804235 16402 leader_election.cc:288] T 0801f66980684fe8baafd65b49b0cd83 P 484d1f530fc44784a3be857749e2aec1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e85b9450f44f4ba097cca79c9d37669c (127.0.97.3:42459), dacd9f26586346c0ab4b9922f88308b3 (127.0.97.1:45245)
I20260620 23:39:24.811600 16391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:24.804788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41015 (local address 127.0.97.3:42459)
0620 23:39:24.805074 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0620 23:39:24.805082 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:39:24.805535 (+ 453us) server_negotiation.cc:407] Connection header received
0620 23:39:24.805780 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:39:24.805786 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:39:24.805869 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:39:24.805945 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:39:24.806785 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:24.807785 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:24.809721 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:24.810557 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:24.810660 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:24.810743 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 23:39:24.811177 (+ 434us) server_negotiation.cc:299] Negotiation successful
0620 23:39:24.811322 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260620 23:39:24.811697 16403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:24.804599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42459 (local address 127.0.97.2:41015)
0620 23:39:24.805363 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:24.805407 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 23:39:24.805585 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:39:24.806127 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:39:24.806130 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:39:24.806152 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:39:24.806604 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:39:24.806618 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:39:24.807922 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:39:24.807927 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:39:24.809552 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:39:24.809564 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:39:24.810779 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:39:24.810785 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:39:24.810986 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:24.811037 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 23:39:24.811144 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 23:39:24.811290 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":436,"threads_started":1}
I20260620 23:39:24.812000 16393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:24.805674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38683 (local address 127.0.97.1:45245)
0620 23:39:24.805874 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 23:39:24.805879 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:39:24.806221 (+ 342us) server_negotiation.cc:407] Connection header received
0620 23:39:24.806369 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:39:24.806374 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:39:24.806450 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:39:24.806555 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:39:24.807259 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:24.808061 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:24.809857 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:24.810912 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:24.811070 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:24.811189 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 23:39:24.811576 (+ 387us) server_negotiation.cc:299] Negotiation successful
0620 23:39:24.811728 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 23:39:24.812410 16404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:24.805460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45245 (local address 127.0.97.2:38683)
0620 23:39:24.806107 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:24.806123 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:39:24.806238 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:39:24.806590 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:39:24.806593 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:39:24.806613 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:39:24.807106 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:39:24.807118 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:39:24.808200 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:39:24.808204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:39:24.809699 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:39:24.809714 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:39:24.811181 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:39:24.811188 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:39:24.811373 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:24.811415 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 23:39:24.811528 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 23:39:24.811734 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"mutex_wait_us":5,"thread_start_us":333,"threads_started":1}
I20260620 23:39:24.813076 16330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0801f66980684fe8baafd65b49b0cd83" candidate_uuid: "484d1f530fc44784a3be857749e2aec1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e85b9450f44f4ba097cca79c9d37669c" is_pre_election: true
I20260620 23:39:24.813594 16330 raft_consensus.cc:2454] T 0801f66980684fe8baafd65b49b0cd83 P e85b9450f44f4ba097cca79c9d37669c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 484d1f530fc44784a3be857749e2aec1 in term 0.
I20260620 23:39:24.813431 16062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0801f66980684fe8baafd65b49b0cd83" candidate_uuid: "484d1f530fc44784a3be857749e2aec1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dacd9f26586346c0ab4b9922f88308b3" is_pre_election: true
I20260620 23:39:24.814055 16062 raft_consensus.cc:2454] T 0801f66980684fe8baafd65b49b0cd83 P dacd9f26586346c0ab4b9922f88308b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 484d1f530fc44784a3be857749e2aec1 in term 0.
I20260620 23:39:24.814306 16130 leader_election.cc:302] T 0801f66980684fe8baafd65b49b0cd83 P 484d1f530fc44784a3be857749e2aec1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 484d1f530fc44784a3be857749e2aec1, e85b9450f44f4ba097cca79c9d37669c; no voters:
I20260620 23:39:24.814723 16402 raft_consensus.cc:2781] T 0801f66980684fe8baafd65b49b0cd83 P 484d1f530fc44784a3be857749e2aec1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:39:24.814874 16402 raft_consensus.cc:482] T 0801f66980684fe8baafd65b49b0cd83 P 484d1f530fc44784a3be857749e2aec1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:39:24.815048 16402 raft_consensus.cc:3037] T 0801f66980684fe8baafd65b49b0cd83 P 484d1f530fc44784a3be857749e2aec1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:39:24.817324 16402 raft_consensus.cc:504] T 0801f66980684fe8baafd65b49b0cd83 P 484d1f530fc44784a3be857749e2aec1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484d1f530fc44784a3be857749e2aec1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46215 } } peers { permanent_uuid: "e85b9450f44f4ba097cca79c9d37669c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42459 } } peers { permanent_uuid: "dacd9f26586346c0ab4b9922f88308b3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45245 } }
I20260620 23:39:24.817888 16402 leader_election.cc:288] T 0801f66980684fe8baafd65b49b0cd83 P 484d1f530fc44784a3be857749e2aec1 [CANDIDATE]: Term 1 election: Requested vote from peers e85b9450f44f4ba097cca79c9d37669c (127.0.97.3:42459), dacd9f26586346c0ab4b9922f88308b3 (127.0.97.1:45245)
I20260620 23:39:24.818511 16330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0801f66980684fe8baafd65b49b0cd83" candidate_uuid: "484d1f530fc44784a3be857749e2aec1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e85b9450f44f4ba097cca79c9d37669c"
I20260620 23:39:24.818627 16062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0801f66980684fe8baafd65b49b0cd83" candidate_uuid: "484d1f530fc44784a3be857749e2aec1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dacd9f26586346c0ab4b9922f88308b3"
I20260620 23:39:24.818753 16330 raft_consensus.cc:3037] T 0801f66980684fe8baafd65b49b0cd83 P e85b9450f44f4ba097cca79c9d37669c [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:39:24.818856 16062 raft_consensus.cc:3037] T 0801f66980684fe8baafd65b49b0cd83 P dacd9f26586346c0ab4b9922f88308b3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:39:24.821058 16330 raft_consensus.cc:2454] T 0801f66980684fe8baafd65b49b0cd83 P e85b9450f44f4ba097cca79c9d37669c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 484d1f530fc44784a3be857749e2aec1 in term 1.
I20260620 23:39:24.821070 16062 raft_consensus.cc:2454] T 0801f66980684fe8baafd65b49b0cd83 P dacd9f26586346c0ab4b9922f88308b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 484d1f530fc44784a3be857749e2aec1 in term 1.
I20260620 23:39:24.821506 16130 leader_election.cc:302] T 0801f66980684fe8baafd65b49b0cd83 P 484d1f530fc44784a3be857749e2aec1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 484d1f530fc44784a3be857749e2aec1, e85b9450f44f4ba097cca79c9d37669c; no voters:
I20260620 23:39:24.821870 16402 raft_consensus.cc:2781] T 0801f66980684fe8baafd65b49b0cd83 P 484d1f530fc44784a3be857749e2aec1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:39:24.822335 16402 raft_consensus.cc:686] T 0801f66980684fe8baafd65b49b0cd83 P 484d1f530fc44784a3be857749e2aec1 [term 1 LEADER]: Becoming Leader. State: Replica: 484d1f530fc44784a3be857749e2aec1, State: Running, Role: LEADER
I20260620 23:39:24.822963 16402 consensus_queue.cc:237] T 0801f66980684fe8baafd65b49b0cd83 P 484d1f530fc44784a3be857749e2aec1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484d1f530fc44784a3be857749e2aec1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46215 } } peers { permanent_uuid: "e85b9450f44f4ba097cca79c9d37669c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42459 } } peers { permanent_uuid: "dacd9f26586346c0ab4b9922f88308b3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45245 } }
I20260620 23:39:24.827661 15916 catalog_manager.cc:5495] T 0801f66980684fe8baafd65b49b0cd83 P 484d1f530fc44784a3be857749e2aec1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 484d1f530fc44784a3be857749e2aec1 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "484d1f530fc44784a3be857749e2aec1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484d1f530fc44784a3be857749e2aec1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e85b9450f44f4ba097cca79c9d37669c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dacd9f26586346c0ab4b9922f88308b3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45245 } health_report { overall_health: UNKNOWN } } }
I20260620 23:39:24.849852 16389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:24.844072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54178 (local address 127.0.97.2:46215)
0620 23:39:24.844261 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 23:39:24.844265 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:39:24.844283 (+ 18us) server_negotiation.cc:407] Connection header received
0620 23:39:24.844449 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:39:24.844453 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:39:24.844533 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:39:24.844666 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:39:24.845032 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:24.845788 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:24.847579 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:24.848172 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:24.848276 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:24.848780 (+ 504us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:39:24.849315 (+ 535us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:39:24.849434 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 23:39:24.849562 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 23:39:24.849633 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 23:39:24.858633 16062 raft_consensus.cc:1261] T 0801f66980684fe8baafd65b49b0cd83 P dacd9f26586346c0ab4b9922f88308b3 [term 1 FOLLOWER]: Refusing update from remote peer 484d1f530fc44784a3be857749e2aec1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:39:24.858642 16330 raft_consensus.cc:1261] T 0801f66980684fe8baafd65b49b0cd83 P e85b9450f44f4ba097cca79c9d37669c [term 1 FOLLOWER]: Refusing update from remote peer 484d1f530fc44784a3be857749e2aec1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:39:24.859753 16402 consensus_queue.cc:1035] T 0801f66980684fe8baafd65b49b0cd83 P 484d1f530fc44784a3be857749e2aec1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dacd9f26586346c0ab4b9922f88308b3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:39:24.860078 16405 consensus_queue.cc:1035] T 0801f66980684fe8baafd65b49b0cd83 P 484d1f530fc44784a3be857749e2aec1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e85b9450f44f4ba097cca79c9d37669c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:39:24.870414 16408 mvcc.cc:205] Tried to move back new op lower bound from 7299066940849664000 to 7299066940719022080. Current Snapshot: MvccSnapshot[applied={T|T < 7299066940849664000}]
I20260620 23:39:24.887990 15917 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:39:24.888360 15917 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:39:24.893105 15917 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b891c2846180438488602be42d467117: Sending DeleteTablet for 3 replicas of tablet 0801f66980684fe8baafd65b49b0cd83
I20260620 23:39:24.894479 16176 tablet_service.cc:1514] Processing DeleteTablet for tablet 0801f66980684fe8baafd65b49b0cd83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:39:24 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54176
I20260620 23:39:24.895136 16310 tablet_service.cc:1514] Processing DeleteTablet for tablet 0801f66980684fe8baafd65b49b0cd83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:39:24 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56564
I20260620 23:39:24.898957 16042 tablet_service.cc:1514] Processing DeleteTablet for tablet 0801f66980684fe8baafd65b49b0cd83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:39:24 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34370
I20260620 23:39:24.900557 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 15979
I20260620 23:39:24.902362 16419 tablet_replica.cc:330] T 0801f66980684fe8baafd65b49b0cd83 P e85b9450f44f4ba097cca79c9d37669c: stopping tablet replica
I20260620 23:39:24.902740 16419 raft_consensus.cc:2229] T 0801f66980684fe8baafd65b49b0cd83 P e85b9450f44f4ba097cca79c9d37669c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:39:24.903218 16419 raft_consensus.cc:2258] T 0801f66980684fe8baafd65b49b0cd83 P e85b9450f44f4ba097cca79c9d37669c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:39:24.906301 16420 tablet_replica.cc:330] T 0801f66980684fe8baafd65b49b0cd83 P dacd9f26586346c0ab4b9922f88308b3: stopping tablet replica
I20260620 23:39:24.906767 16420 raft_consensus.cc:2229] T 0801f66980684fe8baafd65b49b0cd83 P dacd9f26586346c0ab4b9922f88308b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:39:24.907373 16420 raft_consensus.cc:2258] T 0801f66980684fe8baafd65b49b0cd83 P dacd9f26586346c0ab4b9922f88308b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:39:24.907878 16419 ts_tablet_manager.cc:1905] T 0801f66980684fe8baafd65b49b0cd83 P e85b9450f44f4ba097cca79c9d37669c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:39:24.910612 16420 ts_tablet_manager.cc:1905] T 0801f66980684fe8baafd65b49b0cd83 P dacd9f26586346c0ab4b9922f88308b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:39:24.912137 16418 tablet_replica.cc:330] T 0801f66980684fe8baafd65b49b0cd83 P 484d1f530fc44784a3be857749e2aec1: stopping tablet replica
I20260620 23:39:24.912642 16418 raft_consensus.cc:2229] T 0801f66980684fe8baafd65b49b0cd83 P 484d1f530fc44784a3be857749e2aec1 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:39:24.913430 16418 raft_consensus.cc:2258] T 0801f66980684fe8baafd65b49b0cd83 P 484d1f530fc44784a3be857749e2aec1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:39:24.915390 16419 ts_tablet_manager.cc:1918] T 0801f66980684fe8baafd65b49b0cd83 P e85b9450f44f4ba097cca79c9d37669c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:39:24.915582 16419 log.cc:1196] T 0801f66980684fe8baafd65b49b0cd83 P e85b9450f44f4ba097cca79c9d37669c: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-2/wal/wals/0801f66980684fe8baafd65b49b0cd83
I20260620 23:39:24.916165 16419 ts_tablet_manager.cc:1939] T 0801f66980684fe8baafd65b49b0cd83 P e85b9450f44f4ba097cca79c9d37669c: Deleting consensus metadata
I20260620 23:39:24.916797 16418 ts_tablet_manager.cc:1905] T 0801f66980684fe8baafd65b49b0cd83 P 484d1f530fc44784a3be857749e2aec1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:39:24.918247 15903 catalog_manager.cc:4841] TS e85b9450f44f4ba097cca79c9d37669c (127.0.97.3:42459): tablet 0801f66980684fe8baafd65b49b0cd83 (table test-table [id=9ebdaeb759024a6d8ed6855ae1d35910]) successfully deleted
Jun 20 23:39:24 dist-test-slave-tcgx krb5kdc[15873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 23:39:24.922257 16420 ts_tablet_manager.cc:1918] T 0801f66980684fe8baafd65b49b0cd83 P dacd9f26586346c0ab4b9922f88308b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:39:24.922577 16420 log.cc:1196] T 0801f66980684fe8baafd65b49b0cd83 P dacd9f26586346c0ab4b9922f88308b3: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-0/wal/wals/0801f66980684fe8baafd65b49b0cd83
I20260620 23:39:24.923321 16420 ts_tablet_manager.cc:1939] T 0801f66980684fe8baafd65b49b0cd83 P dacd9f26586346c0ab4b9922f88308b3: Deleting consensus metadata
I20260620 23:39:24.924957 16418 ts_tablet_manager.cc:1918] T 0801f66980684fe8baafd65b49b0cd83 P 484d1f530fc44784a3be857749e2aec1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:39:24.925158 16418 log.cc:1196] T 0801f66980684fe8baafd65b49b0cd83 P 484d1f530fc44784a3be857749e2aec1: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-1/wal/wals/0801f66980684fe8baafd65b49b0cd83
I20260620 23:39:24.925776 16418 ts_tablet_manager.cc:1939] T 0801f66980684fe8baafd65b49b0cd83 P 484d1f530fc44784a3be857749e2aec1: Deleting consensus metadata
I20260620 23:39:24.926589 15903 catalog_manager.cc:4841] TS dacd9f26586346c0ab4b9922f88308b3 (127.0.97.1:45245): tablet 0801f66980684fe8baafd65b49b0cd83 (table test-table [id=9ebdaeb759024a6d8ed6855ae1d35910]) successfully deleted
I20260620 23:39:24.927752 15904 catalog_manager.cc:4841] TS 484d1f530fc44784a3be857749e2aec1 (127.0.97.2:46215): tablet 0801f66980684fe8baafd65b49b0cd83 (table test-table [id=9ebdaeb759024a6d8ed6855ae1d35910]) successfully deleted
I20260620 23:39:24.930984 16393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:24.901902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34384 (local address 127.0.97.1:45245)
0620 23:39:24.902123 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 23:39:24.902131 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:39:24.905503 (+ 3372us) server_negotiation.cc:407] Connection header received
0620 23:39:24.905741 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:39:24.905753 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:39:24.905886 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:39:24.905991 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:39:24.908242 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:24.909421 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:24.911816 (+ 2395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:24.912493 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:24.912764 (+ 271us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:24.922882 (+ 10118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:39:24.922904 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:39:24.922928 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:39:24.923028 (+ 100us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:39:24.927434 (+ 4406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:39:24.928388 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:39:24.928396 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:39:24.928398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:39:24.928540 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:39:24.929060 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:39:24.929066 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:39:24.929070 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:39:24.929594 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:39:24.929958 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0620 23:39:24.930227 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 23:39:24.930490 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 23:39:24.931834 16102 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:39:25.034584 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 15979
I20260620 23:39:25.060165 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 16113
Jun 20 23:39:25 dist-test-slave-tcgx krb5kdc[15873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 23:39:25.082775 16389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:25.061429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54190 (local address 127.0.97.2:46215)
0620 23:39:25.061649 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 23:39:25.061656 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:39:25.061680 (+ 24us) server_negotiation.cc:407] Connection header received
0620 23:39:25.061853 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:39:25.061859 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:39:25.061951 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:39:25.062066 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:39:25.064075 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:25.065496 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:25.067795 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:25.068564 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:25.068704 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:25.074679 (+ 5975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:39:25.074706 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:39:25.074730 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:39:25.074798 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:39:25.079357 (+ 4559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:39:25.080138 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:39:25.080149 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:39:25.080154 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:39:25.080366 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:39:25.080874 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:39:25.080880 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:39:25.080884 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:39:25.081395 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:39:25.081792 (+ 397us) server_negotiation.cc:1035] Waiting for connection context
0620 23:39:25.082046 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 23:39:25.082302 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 23:39:25.083616 16236 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:39:25.197220 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 16113
I20260620 23:39:25.217806 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 16247
Jun 20 23:39:25 dist-test-slave-tcgx krb5kdc[15873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 23:39:25.233744 16391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:25.218578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56578 (local address 127.0.97.3:42459)
0620 23:39:25.218772 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 23:39:25.218780 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:39:25.218818 (+ 38us) server_negotiation.cc:407] Connection header received
0620 23:39:25.219074 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:39:25.219079 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:39:25.219147 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:39:25.219252 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:39:25.220772 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:25.221496 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:25.222779 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:25.223435 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:25.223571 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:25.227989 (+ 4418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:39:25.228004 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:39:25.228020 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:39:25.228064 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:39:25.231053 (+ 2989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:39:25.231751 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:39:25.231758 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:39:25.231761 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:39:25.231908 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:39:25.232337 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:39:25.232340 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:39:25.232342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:39:25.232697 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:39:25.232907 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 23:39:25.233240 (+ 333us) server_negotiation.cc:299] Negotiation successful
0620 23:39:25.233402 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 23:39:25.234493 16370 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:39:25.325215 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 16247
I20260620 23:39:25.342687 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 15886
I20260620 23:39:25.344095 15948 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:39:25.457458 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 15886
I20260620 23:39:45.481060 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37589
--webserver_interface=127.0.97.62
--webserver_port=35631
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37589
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:39:45.827379 16426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:39:45.827678 16426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:39:45.827869 16426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:39:45.835299 16426 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 23:39:45.835404 16426 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 23:39:45.835484 16426 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 23:39:45.835565 16426 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:39:45.835670 16426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:39:45.835769 16426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:39:45.835855 16426 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:39:45.835932 16426 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:39:45.846529 16426 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37589
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=35631
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:39:45.847980 16426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:39:45.849476 16426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:39:45.873301 16432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:39:45.873205 16433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:39:45.873486 16426 server_base.cc:1034] running on GCE node
W20260620 23:39:45.873250 16435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:39:45.874884 16426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:39:45.876681 16426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:39:45.877938 16426 hybrid_clock.cc:648] HybridClock initialized: now 1781998785877860 us; error 54 us; skew 500 ppm
I20260620 23:39:45.878397 16426 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:39:45 dist-test-slave-tcgx krb5kdc[15873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998785, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:39:45.882066 16426 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 23:39:45.883908 16426 webserver.cc:460] Webserver started at http://127.0.97.62:35631/ using document root <none> and password file <none>
I20260620 23:39:45.884747 16426 fs_manager.cc:359] Metadata directory not provided
I20260620 23:39:45.884898 16426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:39:45.891505 16426 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 23:39:45.894498 16442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:39:45.896103 16426 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:39:45.896301 16426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/master-0/wal
uuid: "b891c2846180438488602be42d467117"
format_stamp: "Formatted at 2026-06-20 23:39:22 on dist-test-slave-tcgx"
server_key: "e941f9af04805e6454aa55d5cd65b181"
server_key_iv: "214e658b56ac36088703a664b5f2dbd8"
server_key_version: "encryptionkey@0"
I20260620 23:39:45.897434 16426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:39:45.917997 16426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:39:45.922892 16426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:39:45.923316 16426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:39:45.943713 16426 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37589
I20260620 23:39:45.943751 16493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37589 every 8 connection(s)
I20260620 23:39:45.945358 16426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:39:45.947463 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 16426
I20260620 23:39:45.948531 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:45245
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:37589
--webserver_port=42213
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:39:45.955960 16494 sys_catalog.cc:263] Verifying existing consensus state
I20260620 23:39:45.960095 16494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b891c2846180438488602be42d467117: Bootstrap starting.
I20260620 23:39:45.980806 16494 log.cc:824] T 00000000000000000000000000000000 P b891c2846180438488602be42d467117: Log is configured to *not* fsync() on all Append() calls
I20260620 23:39:45.993481 16494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b891c2846180438488602be42d467117: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:39:45.994095 16494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b891c2846180438488602be42d467117: Bootstrap complete.
I20260620 23:39:46.004154 16494 raft_consensus.cc:348] T 00000000000000000000000000000000 P b891c2846180438488602be42d467117 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b891c2846180438488602be42d467117" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37589 } }
I20260620 23:39:46.005048 16494 raft_consensus.cc:729] T 00000000000000000000000000000000 P b891c2846180438488602be42d467117 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b891c2846180438488602be42d467117, State: Initialized, Role: FOLLOWER
I20260620 23:39:46.005756 16494 consensus_queue.cc:260] T 00000000000000000000000000000000 P b891c2846180438488602be42d467117 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b891c2846180438488602be42d467117" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37589 } }
I20260620 23:39:46.006007 16494 raft_consensus.cc:388] T 00000000000000000000000000000000 P b891c2846180438488602be42d467117 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:39:46.006146 16494 raft_consensus.cc:482] T 00000000000000000000000000000000 P b891c2846180438488602be42d467117 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:39:46.006335 16494 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b891c2846180438488602be42d467117 [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:39:46.010515 16494 raft_consensus.cc:504] T 00000000000000000000000000000000 P b891c2846180438488602be42d467117 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b891c2846180438488602be42d467117" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37589 } }
I20260620 23:39:46.011126 16494 leader_election.cc:302] T 00000000000000000000000000000000 P b891c2846180438488602be42d467117 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b891c2846180438488602be42d467117; no voters:
I20260620 23:39:46.011817 16494 leader_election.cc:288] T 00000000000000000000000000000000 P b891c2846180438488602be42d467117 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 23:39:46.011963 16498 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b891c2846180438488602be42d467117 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 23:39:46.013209 16498 raft_consensus.cc:686] T 00000000000000000000000000000000 P b891c2846180438488602be42d467117 [term 2 LEADER]: Becoming Leader. State: Replica: b891c2846180438488602be42d467117, State: Running, Role: LEADER
I20260620 23:39:46.013863 16498 consensus_queue.cc:237] T 00000000000000000000000000000000 P b891c2846180438488602be42d467117 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b891c2846180438488602be42d467117" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37589 } }
I20260620 23:39:46.014115 16494 sys_catalog.cc:564] T 00000000000000000000000000000000 P b891c2846180438488602be42d467117 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:39:46.016294 16500 sys_catalog.cc:455] T 00000000000000000000000000000000 P b891c2846180438488602be42d467117 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b891c2846180438488602be42d467117. Latest consensus state: current_term: 2 leader_uuid: "b891c2846180438488602be42d467117" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b891c2846180438488602be42d467117" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37589 } } }
I20260620 23:39:46.016665 16500 sys_catalog.cc:458] T 00000000000000000000000000000000 P b891c2846180438488602be42d467117 [sys.catalog]: This master's current role is: LEADER
I20260620 23:39:46.017203 16499 sys_catalog.cc:455] T 00000000000000000000000000000000 P b891c2846180438488602be42d467117 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b891c2846180438488602be42d467117" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b891c2846180438488602be42d467117" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37589 } } }
I20260620 23:39:46.017503 16499 sys_catalog.cc:458] T 00000000000000000000000000000000 P b891c2846180438488602be42d467117 [sys.catalog]: This master's current role is: LEADER
I20260620 23:39:46.020972 16504 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:39:46.031052 16504 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:39:46.033032 16504 catalog_manager.cc:1259] Loaded cluster ID: 6bfd6b9c73a3459584b5ce669768df17
I20260620 23:39:46.033239 16504 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:39:46.038683 16504 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:39:46.040827 16504 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b891c2846180438488602be42d467117: Loaded TSK: 0
I20260620 23:39:46.049575 16515 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 23:39:46.081184 16504 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b891c2846180438488602be42d467117: Generated new TSK 1
I20260620 23:39:46.081676 16504 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 23:39:46.310225 16496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:39:46.310585 16496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:39:46.310765 16496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:39:46.318660 16496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:39:46.318782 16496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:39:46.319025 16496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 23:39:46.329586 16496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:45245
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=42213
--tserver_master_addrs=127.0.97.62:37589
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:39:46.331070 16496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:39:46.332454 16496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:39:46.347904 16521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:39:46.347934 16524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:39:46.348557 16522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:39:46.349422 16496 server_base.cc:1034] running on GCE node
I20260620 23:39:46.350080 16496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:39:46.351224 16496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:39:46.352504 16496 hybrid_clock.cc:648] HybridClock initialized: now 1781998786352447 us; error 45 us; skew 500 ppm
I20260620 23:39:46.352892 16496 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:39:46 dist-test-slave-tcgx krb5kdc[15873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998786, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:39:46.356660 16496 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 23:39:46.358422 16496 webserver.cc:460] Webserver started at http://127.0.97.1:42213/ using document root <none> and password file <none>
I20260620 23:39:46.359256 16496 fs_manager.cc:359] Metadata directory not provided
I20260620 23:39:46.359416 16496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:39:46.365130 16496 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 23:39:46.368124 16531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:39:46.369637 16496 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:39:46.369812 16496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-0/wal
uuid: "dacd9f26586346c0ab4b9922f88308b3"
format_stamp: "Formatted at 2026-06-20 23:39:23 on dist-test-slave-tcgx"
server_key: "4ddd36bdf282d4bd73ad7efdc123b3f6"
server_key_iv: "a80e365783dc04e479250ba41387a93c"
server_key_version: "encryptionkey@0"
I20260620 23:39:46.370824 16496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:39:46.390240 16496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:39:46.394753 16496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:39:46.395066 16496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:39:46.396214 16496 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:39:46.397683 16496 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:39:46.397775 16496 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:39:46.397929 16496 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:39:46.398029 16496 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:39:46.432705 16496 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45245
I20260620 23:39:46.432816 16643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45245 every 8 connection(s)
I20260620 23:39:46.434252 16496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
I20260620 23:39:46.436337 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 16496
I20260620 23:39:46.437023 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:46215
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:37589
--webserver_port=33373
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:39:46 dist-test-slave-tcgx krb5kdc[15873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998786, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:39:46.463414 16647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:46.437854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37589 (local address 127.0.97.1:41539)
0620 23:39:46.438817 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:46.438870 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 23:39:46.441657 (+ 2787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:39:46.443280 (+ 1623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:39:46.443293 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:39:46.444133 (+ 840us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:39:46.444957 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:39:46.444986 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:39:46.446692 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:39:46.446696 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:39:46.448071 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:39:46.448086 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:39:46.449731 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:39:46.449736 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:39:46.449964 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:46.450717 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:39:46.450741 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:39:46.453495 (+ 2754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:39:46.458794 (+ 5299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:39:46.458801 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:39:46.458815 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:39:46.459379 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:39:46.459956 (+ 577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:39:46.459960 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:39:46.459962 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:39:46.460145 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:39:46.461263 (+ 1118us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:39:46.461269 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:39:46.461585 (+ 316us) client_negotiation.cc:769] Sending connection context
0620 23:39:46.461830 (+ 245us) client_negotiation.cc:240] Negotiation successful
0620 23:39:46.462260 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":389,"threads_started":1}
I20260620 23:39:46.464082 16646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:46.437702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41539 (local address 127.0.97.62:37589)
0620 23:39:46.438442 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0620 23:39:46.438460 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 23:39:46.439029 (+ 569us) server_negotiation.cc:407] Connection header received
0620 23:39:46.442321 (+ 3292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:39:46.442337 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:39:46.442709 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:39:46.443109 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:39:46.445227 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:46.446484 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:46.448236 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:46.449460 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:46.449675 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:46.453847 (+ 4172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:39:46.453862 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:39:46.453887 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:39:46.453945 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:39:46.458595 (+ 4650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:39:46.459549 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:39:46.459561 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:39:46.459573 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:39:46.459824 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:39:46.460310 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:39:46.460318 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:39:46.460323 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:39:46.461080 (+ 757us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:39:46.461331 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0620 23:39:46.462145 (+ 814us) server_negotiation.cc:299] Negotiation successful
0620 23:39:46.462657 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":278,"threads_started":1}
I20260620 23:39:46.467243 16644 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37589
I20260620 23:39:46.467634 16644 heartbeater.cc:464] Registering TS with master...
I20260620 23:39:46.468518 16644 heartbeater.cc:511] Master 127.0.97.62:37589 requested a full tablet report, sending...
I20260620 23:39:46.471208 16459 ts_manager.cc:194] Registered new tserver with Master: dacd9f26586346c0ab4b9922f88308b3 (127.0.97.1:45245)
I20260620 23:39:46.473520 16459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:41539
W20260620 23:39:46.788028 16648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:39:46.788328 16648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:39:46.788430 16648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:39:46.795958 16648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:39:46.796078 16648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:39:46.796195 16648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 23:39:46.806501 16648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:46215
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=33373
--tserver_master_addrs=127.0.97.62:37589
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:39:46.807927 16648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:39:46.809224 16648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:39:46.822798 16656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:39:46.822827 16654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:39:46.823253 16653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:39:46.824352 16648 server_base.cc:1034] running on GCE node
I20260620 23:39:46.824949 16648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:39:46.825915 16648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:39:46.827124 16648 hybrid_clock.cc:648] HybridClock initialized: now 1781998786827046 us; error 75 us; skew 500 ppm
I20260620 23:39:46.827495 16648 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:39:46 dist-test-slave-tcgx krb5kdc[15873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998786, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:39:46.830845 16648 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 23:39:46.832547 16648 webserver.cc:460] Webserver started at http://127.0.97.2:33373/ using document root <none> and password file <none>
I20260620 23:39:46.833287 16648 fs_manager.cc:359] Metadata directory not provided
I20260620 23:39:46.833454 16648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:39:46.839150 16648 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 23:39:46.841928 16663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:39:46.843519 16648 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:39:46.843724 16648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-1/wal
uuid: "484d1f530fc44784a3be857749e2aec1"
format_stamp: "Formatted at 2026-06-20 23:39:23 on dist-test-slave-tcgx"
server_key: "3f903f91fa429b32b3990ca10bc07ecc"
server_key_iv: "78277e1329cf21e3a17e33fad220cc3e"
server_key_version: "encryptionkey@0"
I20260620 23:39:46.844794 16648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:39:46.860212 16648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:39:46.864611 16648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:39:46.864909 16648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:39:46.865897 16648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:39:46.867336 16648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:39:46.867457 16648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:39:46.867591 16648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:39:46.867681 16648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:39:46.900971 16648 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46215
I20260620 23:39:46.901060 16775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46215 every 8 connection(s)
I20260620 23:39:46.902761 16648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
I20260620 23:39:46.906579 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 16648
I20260620 23:39:46.907464 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:42459
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:37589
--webserver_port=35389
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:39:46 dist-test-slave-tcgx krb5kdc[15873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998786, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:39:46.925637 16646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:46.905575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57519 (local address 127.0.97.62:37589)
0620 23:39:46.905789 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 23:39:46.905794 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:39:46.907059 (+ 1265us) server_negotiation.cc:407] Connection header received
0620 23:39:46.909065 (+ 2006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:39:46.909075 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:39:46.909182 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:39:46.909318 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:39:46.910896 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:46.912325 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:46.914642 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:46.915150 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:46.915306 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:46.919023 (+ 3717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:39:46.919031 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:39:46.919036 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:39:46.919081 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:39:46.922099 (+ 3018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:39:46.923008 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:39:46.923017 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:39:46.923021 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:39:46.923173 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:39:46.923674 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:39:46.923682 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:39:46.923686 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:39:46.924005 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:39:46.924176 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 23:39:46.924969 (+ 793us) server_negotiation.cc:299] Negotiation successful
0620 23:39:46.925251 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 23:39:46.926725 16778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:46.906006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37589 (local address 127.0.97.2:57519)
0620 23:39:46.906842 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:46.906897 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 23:39:46.908758 (+ 1861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:39:46.909532 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:39:46.909541 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:39:46.910091 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:39:46.910689 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:39:46.910707 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:39:46.912527 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:39:46.912535 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:39:46.914454 (+ 1919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:39:46.914472 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:39:46.915399 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:39:46.915406 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:39:46.915747 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:46.916486 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:39:46.916510 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:39:46.918703 (+ 2193us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:39:46.922291 (+ 3588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:39:46.922302 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:39:46.922324 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:39:46.922835 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:39:46.923319 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:39:46.923325 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:39:46.923328 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:39:46.923540 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:39:46.924159 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:39:46.924168 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:39:46.924557 (+ 389us) client_negotiation.cc:769] Sending connection context
0620 23:39:46.924881 (+ 324us) client_negotiation.cc:240] Negotiation successful
0620 23:39:46.925491 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":358,"threads_started":1}
I20260620 23:39:46.928478 16776 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37589
I20260620 23:39:46.928891 16776 heartbeater.cc:464] Registering TS with master...
I20260620 23:39:46.929961 16776 heartbeater.cc:511] Master 127.0.97.62:37589 requested a full tablet report, sending...
I20260620 23:39:46.931887 16459 ts_manager.cc:194] Registered new tserver with Master: 484d1f530fc44784a3be857749e2aec1 (127.0.97.2:46215)
I20260620 23:39:46.932777 16459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:57519
W20260620 23:39:47.256644 16779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:39:47.256940 16779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:39:47.257113 16779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:39:47.264603 16779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:39:47.264751 16779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:39:47.264928 16779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 23:39:47.275218 16779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:42459
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=35389
--tserver_master_addrs=127.0.97.62:37589
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:39:47.276644 16779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:39:47.278004 16779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:39:47.291489 16784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:39:47.291626 16785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:39:47.291630 16787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:39:47.292899 16779 server_base.cc:1034] running on GCE node
I20260620 23:39:47.293576 16779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:39:47.294646 16779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:39:47.295910 16779 hybrid_clock.cc:648] HybridClock initialized: now 1781998787295834 us; error 71 us; skew 500 ppm
I20260620 23:39:47.296298 16779 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:39:47 dist-test-slave-tcgx krb5kdc[15873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998787, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:39:47.299679 16779 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 23:39:47.301414 16779 webserver.cc:460] Webserver started at http://127.0.97.3:35389/ using document root <none> and password file <none>
I20260620 23:39:47.302150 16779 fs_manager.cc:359] Metadata directory not provided
I20260620 23:39:47.302313 16779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:39:47.308056 16779 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 23:39:47.311139 16794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:39:47.312577 16779 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 23:39:47.312784 16779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-2/wal
uuid: "e85b9450f44f4ba097cca79c9d37669c"
format_stamp: "Formatted at 2026-06-20 23:39:24 on dist-test-slave-tcgx"
server_key: "e1740287c5d986a49ac6fcf0d0cf1805"
server_key_iv: "fc3d60b0e3ed9d7ffe927cdac0327937"
server_key_version: "encryptionkey@0"
I20260620 23:39:47.313819 16779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:39:47.339097 16779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:39:47.343804 16779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:39:47.344086 16779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:39:47.345095 16779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:39:47.346469 16779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:39:47.346592 16779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:39:47.346726 16779 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:39:47.346812 16779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:39:47.380436 16779 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42459
I20260620 23:39:47.380533 16906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42459 every 8 connection(s)
I20260620 23:39:47.382015 16779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
I20260620 23:39:47.384511 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 16779
Jun 20 23:39:47 dist-test-slave-tcgx krb5kdc[15873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998787, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:39:47.403231 16909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:47.385469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55986 (local address 127.0.97.62:37589)
0620 23:39:47.385999 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0620 23:39:47.386005 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:39:47.386481 (+ 476us) server_negotiation.cc:407] Connection header received
0620 23:39:47.386683 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:39:47.386688 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:39:47.386772 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:39:47.386892 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:39:47.389158 (+ 2266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:47.389954 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:47.392014 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:47.392804 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:47.392936 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:47.396458 (+ 3522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:39:47.396465 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:39:47.396470 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:39:47.396513 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:39:47.399622 (+ 3109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:39:47.400511 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:39:47.400517 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:39:47.400520 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:39:47.400656 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:39:47.401265 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:39:47.401272 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:39:47.401276 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:39:47.401672 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:39:47.401991 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0620 23:39:47.402413 (+ 422us) server_negotiation.cc:299] Negotiation successful
0620 23:39:47.402712 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":417,"thread_start_us":310,"threads_started":1}
I20260620 23:39:47.407546 16646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:47.384960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37067 (local address 127.0.97.62:37589)
0620 23:39:47.385188 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 23:39:47.385193 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:39:47.386857 (+ 1664us) server_negotiation.cc:407] Connection header received
0620 23:39:47.389643 (+ 2786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:39:47.389648 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:39:47.389739 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:39:47.389835 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:39:47.392014 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:47.393177 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:47.395837 (+ 2660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:47.396339 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:47.396492 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:47.400140 (+ 3648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:39:47.400146 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:39:47.400168 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:39:47.400249 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:39:47.403921 (+ 3672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:39:47.404834 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:39:47.404838 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:39:47.404841 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:39:47.404983 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:39:47.405578 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:39:47.405582 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:39:47.405584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:39:47.405992 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:39:47.406113 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 23:39:47.407044 (+ 931us) server_negotiation.cc:299] Negotiation successful
0620 23:39:47.407283 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":82}
I20260620 23:39:47.408530 16911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:47.385681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37589 (local address 127.0.97.3:37067)
0620 23:39:47.386711 (+ 1030us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:47.386751 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 23:39:47.389357 (+ 2606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:39:47.390138 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:39:47.390150 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:39:47.391009 (+ 859us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:39:47.391833 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:39:47.391857 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:39:47.393354 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:39:47.393359 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:39:47.395587 (+ 2228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:39:47.395608 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:39:47.396566 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:39:47.396571 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:39:47.396788 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:47.397486 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:39:47.397509 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:39:47.399854 (+ 2345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:39:47.404065 (+ 4211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:39:47.404077 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:39:47.404098 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:39:47.404700 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:39:47.405105 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:39:47.405110 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:39:47.405113 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:39:47.405386 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:39:47.406133 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:39:47.406144 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:39:47.406533 (+ 389us) client_negotiation.cc:769] Sending connection context
0620 23:39:47.406784 (+ 251us) client_negotiation.cc:240] Negotiation successful
0620 23:39:47.407305 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":694,"thread_start_us":380,"threads_started":1}
I20260620 23:39:47.410178 16907 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37589
I20260620 23:39:47.410535 16907 heartbeater.cc:464] Registering TS with master...
I20260620 23:39:47.411468 16907 heartbeater.cc:511] Master 127.0.97.62:37589 requested a full tablet report, sending...
I20260620 23:39:47.413174 16459 ts_manager.cc:194] Registered new tserver with Master: e85b9450f44f4ba097cca79c9d37669c (127.0.97.3:42459)
I20260620 23:39:47.414227 16459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:37067
I20260620 23:39:47.418705 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:39:47.425565 16646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:47.419790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55996 (local address 127.0.97.62:37589)
0620 23:39:47.419963 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 23:39:47.419968 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:39:47.420526 (+ 558us) server_negotiation.cc:407] Connection header received
0620 23:39:47.420708 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:39:47.420712 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:39:47.420859 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:39:47.420985 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:39:47.421501 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:47.422307 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:47.423796 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:47.424197 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:47.424325 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:47.424823 (+ 498us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:39:47.425044 (+ 221us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 23:39:47.425363 (+ 319us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55996: token expired
Metrics: {"server-negotiator.queue_time_us":62}
W20260620 23:39:47.425670 16646 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55996: token expired
W20260620 23:39:47.426270 16381 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.97.62:37589)
I20260620 23:39:47.439976 16646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:47.426870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56002 (local address 127.0.97.62:37589)
0620 23:39:47.427110 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 23:39:47.427115 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:39:47.427156 (+ 41us) server_negotiation.cc:407] Connection header received
0620 23:39:47.427321 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:39:47.427325 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:39:47.427389 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:39:47.427493 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:39:47.429043 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:47.429783 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:47.431307 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:47.431707 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:47.431845 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:47.434538 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:39:47.434543 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:39:47.434546 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:39:47.434580 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:39:47.437215 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:39:47.437910 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:39:47.437920 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:39:47.437922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:39:47.438037 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:39:47.438403 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:39:47.438409 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:39:47.438412 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:39:47.438738 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:39:47.439011 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0620 23:39:47.439410 (+ 399us) server_negotiation.cc:299] Negotiation successful
0620 23:39:47.439638 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260620 23:39:47.443562 16459 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56002:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:39:47.450889 16459 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:39:47.474298 16919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:47.463187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44062 (local address 127.0.97.3:42459)
0620 23:39:47.465440 (+ 2253us) server_negotiation.cc:206] Beginning negotiation
0620 23:39:47.465447 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:39:47.465475 (+ 28us) server_negotiation.cc:407] Connection header received
0620 23:39:47.465575 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:39:47.465583 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:39:47.465867 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:39:47.466693 (+ 826us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:39:47.467029 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:47.468773 (+ 1744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:47.472040 (+ 3267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:47.473057 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:47.473232 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:47.473476 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0620 23:39:47.473751 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 23:39:47.473918 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2077,"thread_start_us":1929,"threads_started":1}
I20260620 23:39:47.474476 16918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:47.462909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42459 (local address 127.0.0.1:44062)
0620 23:39:47.463815 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:47.463857 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 23:39:47.464026 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:39:47.466107 (+ 2081us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:39:47.466118 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:39:47.466151 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:39:47.466652 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:39:47.466665 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:39:47.468946 (+ 2281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:39:47.468951 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:39:47.471821 (+ 2870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:39:47.471836 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:39:47.473279 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:39:47.473286 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:39:47.473488 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:47.473549 (+ 61us) client_negotiation.cc:769] Sending connection context
0620 23:39:47.473700 (+ 151us) client_negotiation.cc:240] Negotiation successful
0620 23:39:47.473902 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"mutex_wait_us":69,"thread_start_us":2064,"threads_started":1}
I20260620 23:39:47.475414 16920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:47.463637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60830 (local address 127.0.97.2:46215)
0620 23:39:47.464933 (+ 1296us) server_negotiation.cc:206] Beginning negotiation
0620 23:39:47.464945 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:39:47.465397 (+ 452us) server_negotiation.cc:407] Connection header received
0620 23:39:47.465682 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:39:47.465689 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:39:47.465865 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:39:47.466045 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:39:47.467244 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:47.468792 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:47.472288 (+ 3496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:47.473754 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:47.473900 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:47.474113 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 23:39:47.474858 (+ 745us) server_negotiation.cc:299] Negotiation successful
0620 23:39:47.475101 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1129,"thread_start_us":1021,"threads_started":1}
I20260620 23:39:47.476439 16917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:47.463041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46215 (local address 127.0.0.1:60830)
0620 23:39:47.465284 (+ 2243us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:47.465303 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:39:47.465509 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:39:47.466371 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:39:47.466375 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:39:47.466397 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:39:47.467022 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:39:47.467034 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:39:47.468965 (+ 1931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:39:47.468971 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:39:47.472055 (+ 3084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:39:47.472067 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:39:47.474324 (+ 2257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:39:47.474337 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:39:47.474525 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:47.474664 (+ 139us) client_negotiation.cc:769] Sending connection context
0620 23:39:47.474755 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 23:39:47.474952 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2115,"mutex_wait_us":101,"thread_start_us":744,"threads_started":1}
I20260620 23:39:47.477258 16644 heartbeater.cc:503] Master 127.0.97.62:37589 was elected leader, sending a full tablet report...
I20260620 23:39:47.479421 16921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:47.465395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45245 (local address 127.0.0.1:34516)
0620 23:39:47.470366 (+ 4971us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:47.470381 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:39:47.470551 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:39:47.471548 (+ 997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:39:47.471553 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:39:47.471577 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:39:47.472060 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:39:47.472071 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:39:47.473985 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:39:47.474037 (+ 52us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:39:47.476454 (+ 2417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:39:47.476465 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:39:47.478662 (+ 2197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:39:47.478667 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:39:47.478846 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:47.478880 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 23:39:47.479038 (+ 158us) client_negotiation.cc:240] Negotiation successful
0620 23:39:47.479170 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4831,"mutex_wait_us":262,"thread_start_us":4650,"threads_started":1}
I20260620 23:39:47.478891 16842 tablet_service.cc:1467] Processing CreateTablet for tablet cb2776cf410841af944b68e2af842fc7 (DEFAULT_TABLE table=test-table [id=4288458361f345019fd7e6e672c591b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:39:47.480113 16922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:47.465804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34516 (local address 127.0.97.1:45245)
0620 23:39:47.469519 (+ 3715us) server_negotiation.cc:206] Beginning negotiation
0620 23:39:47.469527 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:39:47.470504 (+ 977us) server_negotiation.cc:407] Connection header received
0620 23:39:47.470779 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:39:47.470789 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:39:47.471097 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:39:47.471303 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:39:47.472262 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:47.473830 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:47.476661 (+ 2831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:47.478184 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:47.478305 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:47.478520 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 23:39:47.479696 (+ 1176us) server_negotiation.cc:299] Negotiation successful
0620 23:39:47.479838 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3534,"thread_start_us":3359,"threads_started":1}
I20260620 23:39:47.480420 16842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb2776cf410841af944b68e2af842fc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:39:47.479974 16711 tablet_service.cc:1467] Processing CreateTablet for tablet cb2776cf410841af944b68e2af842fc7 (DEFAULT_TABLE table=test-table [id=4288458361f345019fd7e6e672c591b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:39:47.481271 16711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb2776cf410841af944b68e2af842fc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:39:47.483089 16579 tablet_service.cc:1467] Processing CreateTablet for tablet cb2776cf410841af944b68e2af842fc7 (DEFAULT_TABLE table=test-table [id=4288458361f345019fd7e6e672c591b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:39:47.484172 16579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb2776cf410841af944b68e2af842fc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:39:47.496744 16924 tablet_bootstrap.cc:492] T cb2776cf410841af944b68e2af842fc7 P e85b9450f44f4ba097cca79c9d37669c: Bootstrap starting.
I20260620 23:39:47.497750 16923 tablet_bootstrap.cc:492] T cb2776cf410841af944b68e2af842fc7 P 484d1f530fc44784a3be857749e2aec1: Bootstrap starting.
I20260620 23:39:47.498723 16925 tablet_bootstrap.cc:492] T cb2776cf410841af944b68e2af842fc7 P dacd9f26586346c0ab4b9922f88308b3: Bootstrap starting.
I20260620 23:39:47.501751 16924 tablet_bootstrap.cc:654] T cb2776cf410841af944b68e2af842fc7 P e85b9450f44f4ba097cca79c9d37669c: Neither blocks nor log segments found. Creating new log.
I20260620 23:39:47.502333 16925 tablet_bootstrap.cc:654] T cb2776cf410841af944b68e2af842fc7 P dacd9f26586346c0ab4b9922f88308b3: Neither blocks nor log segments found. Creating new log.
I20260620 23:39:47.502333 16923 tablet_bootstrap.cc:654] T cb2776cf410841af944b68e2af842fc7 P 484d1f530fc44784a3be857749e2aec1: Neither blocks nor log segments found. Creating new log.
I20260620 23:39:47.503078 16924 log.cc:824] T cb2776cf410841af944b68e2af842fc7 P e85b9450f44f4ba097cca79c9d37669c: Log is configured to *not* fsync() on all Append() calls
I20260620 23:39:47.503679 16925 log.cc:824] T cb2776cf410841af944b68e2af842fc7 P dacd9f26586346c0ab4b9922f88308b3: Log is configured to *not* fsync() on all Append() calls
I20260620 23:39:47.503692 16923 log.cc:824] T cb2776cf410841af944b68e2af842fc7 P 484d1f530fc44784a3be857749e2aec1: Log is configured to *not* fsync() on all Append() calls
I20260620 23:39:47.508252 16924 tablet_bootstrap.cc:492] T cb2776cf410841af944b68e2af842fc7 P e85b9450f44f4ba097cca79c9d37669c: No bootstrap required, opened a new log
I20260620 23:39:47.508628 16924 ts_tablet_manager.cc:1397] T cb2776cf410841af944b68e2af842fc7 P e85b9450f44f4ba097cca79c9d37669c: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.002s
I20260620 23:39:47.509802 16925 tablet_bootstrap.cc:492] T cb2776cf410841af944b68e2af842fc7 P dacd9f26586346c0ab4b9922f88308b3: No bootstrap required, opened a new log
I20260620 23:39:47.510131 16925 ts_tablet_manager.cc:1397] T cb2776cf410841af944b68e2af842fc7 P dacd9f26586346c0ab4b9922f88308b3: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.004s
I20260620 23:39:47.510581 16923 tablet_bootstrap.cc:492] T cb2776cf410841af944b68e2af842fc7 P 484d1f530fc44784a3be857749e2aec1: No bootstrap required, opened a new log
I20260620 23:39:47.511053 16923 ts_tablet_manager.cc:1397] T cb2776cf410841af944b68e2af842fc7 P 484d1f530fc44784a3be857749e2aec1: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.000s
I20260620 23:39:47.520258 16924 raft_consensus.cc:348] T cb2776cf410841af944b68e2af842fc7 P e85b9450f44f4ba097cca79c9d37669c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e85b9450f44f4ba097cca79c9d37669c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42459 } } peers { permanent_uuid: "484d1f530fc44784a3be857749e2aec1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46215 } } peers { permanent_uuid: "dacd9f26586346c0ab4b9922f88308b3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45245 } }
I20260620 23:39:47.520798 16924 raft_consensus.cc:374] T cb2776cf410841af944b68e2af842fc7 P e85b9450f44f4ba097cca79c9d37669c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:39:47.520938 16924 raft_consensus.cc:729] T cb2776cf410841af944b68e2af842fc7 P e85b9450f44f4ba097cca79c9d37669c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e85b9450f44f4ba097cca79c9d37669c, State: Initialized, Role: FOLLOWER
I20260620 23:39:47.521689 16924 consensus_queue.cc:260] T cb2776cf410841af944b68e2af842fc7 P e85b9450f44f4ba097cca79c9d37669c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e85b9450f44f4ba097cca79c9d37669c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42459 } } peers { permanent_uuid: "484d1f530fc44784a3be857749e2aec1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46215 } } peers { permanent_uuid: "dacd9f26586346c0ab4b9922f88308b3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45245 } }
I20260620 23:39:47.522099 16923 raft_consensus.cc:348] T cb2776cf410841af944b68e2af842fc7 P 484d1f530fc44784a3be857749e2aec1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e85b9450f44f4ba097cca79c9d37669c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42459 } } peers { permanent_uuid: "484d1f530fc44784a3be857749e2aec1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46215 } } peers { permanent_uuid: "dacd9f26586346c0ab4b9922f88308b3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45245 } }
I20260620 23:39:47.522725 16923 raft_consensus.cc:374] T cb2776cf410841af944b68e2af842fc7 P 484d1f530fc44784a3be857749e2aec1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:39:47.522863 16923 raft_consensus.cc:729] T cb2776cf410841af944b68e2af842fc7 P 484d1f530fc44784a3be857749e2aec1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 484d1f530fc44784a3be857749e2aec1, State: Initialized, Role: FOLLOWER
I20260620 23:39:47.522789 16925 raft_consensus.cc:348] T cb2776cf410841af944b68e2af842fc7 P dacd9f26586346c0ab4b9922f88308b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e85b9450f44f4ba097cca79c9d37669c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42459 } } peers { permanent_uuid: "484d1f530fc44784a3be857749e2aec1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46215 } } peers { permanent_uuid: "dacd9f26586346c0ab4b9922f88308b3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45245 } }
I20260620 23:39:47.523114 16907 heartbeater.cc:503] Master 127.0.97.62:37589 was elected leader, sending a full tablet report...
I20260620 23:39:47.523298 16925 raft_consensus.cc:374] T cb2776cf410841af944b68e2af842fc7 P dacd9f26586346c0ab4b9922f88308b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:39:47.523422 16925 raft_consensus.cc:729] T cb2776cf410841af944b68e2af842fc7 P dacd9f26586346c0ab4b9922f88308b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dacd9f26586346c0ab4b9922f88308b3, State: Initialized, Role: FOLLOWER
I20260620 23:39:47.523780 16924 ts_tablet_manager.cc:1428] T cb2776cf410841af944b68e2af842fc7 P e85b9450f44f4ba097cca79c9d37669c: Time spent starting tablet: real 0.015s user 0.008s sys 0.005s
I20260620 23:39:47.523676 16923 consensus_queue.cc:260] T cb2776cf410841af944b68e2af842fc7 P 484d1f530fc44784a3be857749e2aec1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e85b9450f44f4ba097cca79c9d37669c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42459 } } peers { permanent_uuid: "484d1f530fc44784a3be857749e2aec1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46215 } } peers { permanent_uuid: "dacd9f26586346c0ab4b9922f88308b3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45245 } }
I20260620 23:39:47.524120 16925 consensus_queue.cc:260] T cb2776cf410841af944b68e2af842fc7 P dacd9f26586346c0ab4b9922f88308b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e85b9450f44f4ba097cca79c9d37669c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42459 } } peers { permanent_uuid: "484d1f530fc44784a3be857749e2aec1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46215 } } peers { permanent_uuid: "dacd9f26586346c0ab4b9922f88308b3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45245 } }
I20260620 23:39:47.524938 16776 heartbeater.cc:503] Master 127.0.97.62:37589 was elected leader, sending a full tablet report...
I20260620 23:39:47.524896 16930 raft_consensus.cc:482] T cb2776cf410841af944b68e2af842fc7 P 484d1f530fc44784a3be857749e2aec1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:39:47.525233 16930 raft_consensus.cc:504] T cb2776cf410841af944b68e2af842fc7 P 484d1f530fc44784a3be857749e2aec1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e85b9450f44f4ba097cca79c9d37669c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42459 } } peers { permanent_uuid: "484d1f530fc44784a3be857749e2aec1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46215 } } peers { permanent_uuid: "dacd9f26586346c0ab4b9922f88308b3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45245 } }
I20260620 23:39:47.527812 16930 leader_election.cc:288] T cb2776cf410841af944b68e2af842fc7 P 484d1f530fc44784a3be857749e2aec1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e85b9450f44f4ba097cca79c9d37669c (127.0.97.3:42459), dacd9f26586346c0ab4b9922f88308b3 (127.0.97.1:45245)
I20260620 23:39:47.529438 16923 ts_tablet_manager.cc:1428] T cb2776cf410841af944b68e2af842fc7 P 484d1f530fc44784a3be857749e2aec1: Time spent starting tablet: real 0.018s user 0.012s sys 0.000s
I20260620 23:39:47.534653 16925 ts_tablet_manager.cc:1428] T cb2776cf410841af944b68e2af842fc7 P dacd9f26586346c0ab4b9922f88308b3: Time spent starting tablet: real 0.024s user 0.019s sys 0.000s
I20260620 23:39:47.539218 16919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:47.528513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51887 (local address 127.0.97.3:42459)
0620 23:39:47.528690 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 23:39:47.528696 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:39:47.532275 (+ 3579us) server_negotiation.cc:407] Connection header received
0620 23:39:47.532343 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:39:47.532348 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:39:47.532423 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:39:47.532512 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:39:47.533347 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:47.534442 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:47.537043 (+ 2601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:47.538225 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:47.538330 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:47.538404 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 23:39:47.538863 (+ 459us) server_negotiation.cc:299] Negotiation successful
0620 23:39:47.539030 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 23:39:47.539549 16933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:47.527962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42459 (local address 127.0.97.2:51887)
0620 23:39:47.531312 (+ 3350us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:47.531340 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 23:39:47.531523 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:39:47.532704 (+ 1181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:39:47.532709 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:39:47.532731 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:39:47.533140 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:39:47.533165 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:39:47.534606 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:39:47.534611 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:39:47.536877 (+ 2266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:39:47.536893 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:39:47.538464 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:39:47.538471 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:39:47.538686 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:47.538726 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 23:39:47.538863 (+ 137us) client_negotiation.cc:240] Negotiation successful
0620 23:39:47.539049 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3164,"thread_start_us":3012,"threads_started":1}
I20260620 23:39:47.540343 16922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:47.532004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47371 (local address 127.0.97.1:45245)
0620 23:39:47.532227 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 23:39:47.532233 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:39:47.532994 (+ 761us) server_negotiation.cc:407] Connection header received
0620 23:39:47.533258 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:39:47.533263 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:39:47.533333 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:39:47.533392 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:39:47.534159 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:47.535273 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:47.538082 (+ 2809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:47.539472 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:47.539640 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:47.539760 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 23:39:47.539968 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 23:39:47.540096 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 23:39:47.540376 16934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:47.531497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45245 (local address 127.0.97.2:47371)
0620 23:39:47.532890 (+ 1393us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:47.532914 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 23:39:47.533096 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:39:47.533578 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:39:47.533582 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:39:47.533606 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:39:47.533952 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:39:47.533963 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:39:47.535421 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:39:47.535424 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:39:47.537880 (+ 2456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:39:47.537897 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:39:47.539674 (+ 1777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:39:47.539678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:39:47.539813 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:47.539835 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 23:39:47.539943 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 23:39:47.540068 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1231,"thread_start_us":4152,"threads_started":1}
I20260620 23:39:47.540846 16862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb2776cf410841af944b68e2af842fc7" candidate_uuid: "484d1f530fc44784a3be857749e2aec1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e85b9450f44f4ba097cca79c9d37669c" is_pre_election: true
I20260620 23:39:47.541383 16862 raft_consensus.cc:2454] T cb2776cf410841af944b68e2af842fc7 P e85b9450f44f4ba097cca79c9d37669c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 484d1f530fc44784a3be857749e2aec1 in term 0.
I20260620 23:39:47.541513 16599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb2776cf410841af944b68e2af842fc7" candidate_uuid: "484d1f530fc44784a3be857749e2aec1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dacd9f26586346c0ab4b9922f88308b3" is_pre_election: true
I20260620 23:39:47.541917 16599 raft_consensus.cc:2454] T cb2776cf410841af944b68e2af842fc7 P dacd9f26586346c0ab4b9922f88308b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 484d1f530fc44784a3be857749e2aec1 in term 0.
I20260620 23:39:47.542150 16665 leader_election.cc:302] T cb2776cf410841af944b68e2af842fc7 P 484d1f530fc44784a3be857749e2aec1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 484d1f530fc44784a3be857749e2aec1, e85b9450f44f4ba097cca79c9d37669c; no voters:
I20260620 23:39:47.542600 16932 raft_consensus.cc:2781] T cb2776cf410841af944b68e2af842fc7 P 484d1f530fc44784a3be857749e2aec1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:39:47.542747 16932 raft_consensus.cc:482] T cb2776cf410841af944b68e2af842fc7 P 484d1f530fc44784a3be857749e2aec1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:39:47.542985 16932 raft_consensus.cc:3037] T cb2776cf410841af944b68e2af842fc7 P 484d1f530fc44784a3be857749e2aec1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:39:47.545818 16932 raft_consensus.cc:504] T cb2776cf410841af944b68e2af842fc7 P 484d1f530fc44784a3be857749e2aec1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e85b9450f44f4ba097cca79c9d37669c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42459 } } peers { permanent_uuid: "484d1f530fc44784a3be857749e2aec1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46215 } } peers { permanent_uuid: "dacd9f26586346c0ab4b9922f88308b3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45245 } }
I20260620 23:39:47.546406 16932 leader_election.cc:288] T cb2776cf410841af944b68e2af842fc7 P 484d1f530fc44784a3be857749e2aec1 [CANDIDATE]: Term 1 election: Requested vote from peers e85b9450f44f4ba097cca79c9d37669c (127.0.97.3:42459), dacd9f26586346c0ab4b9922f88308b3 (127.0.97.1:45245)
I20260620 23:39:47.546917 16862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb2776cf410841af944b68e2af842fc7" candidate_uuid: "484d1f530fc44784a3be857749e2aec1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e85b9450f44f4ba097cca79c9d37669c"
I20260620 23:39:47.547013 16599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb2776cf410841af944b68e2af842fc7" candidate_uuid: "484d1f530fc44784a3be857749e2aec1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dacd9f26586346c0ab4b9922f88308b3"
I20260620 23:39:47.547228 16599 raft_consensus.cc:3037] T cb2776cf410841af944b68e2af842fc7 P dacd9f26586346c0ab4b9922f88308b3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:39:47.547214 16862 raft_consensus.cc:3037] T cb2776cf410841af944b68e2af842fc7 P e85b9450f44f4ba097cca79c9d37669c [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:39:47.549643 16599 raft_consensus.cc:2454] T cb2776cf410841af944b68e2af842fc7 P dacd9f26586346c0ab4b9922f88308b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 484d1f530fc44784a3be857749e2aec1 in term 1.
I20260620 23:39:47.550014 16862 raft_consensus.cc:2454] T cb2776cf410841af944b68e2af842fc7 P e85b9450f44f4ba097cca79c9d37669c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 484d1f530fc44784a3be857749e2aec1 in term 1.
I20260620 23:39:47.550076 16665 leader_election.cc:302] T cb2776cf410841af944b68e2af842fc7 P 484d1f530fc44784a3be857749e2aec1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 484d1f530fc44784a3be857749e2aec1, dacd9f26586346c0ab4b9922f88308b3; no voters:
I20260620 23:39:47.550375 16932 raft_consensus.cc:2781] T cb2776cf410841af944b68e2af842fc7 P 484d1f530fc44784a3be857749e2aec1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:39:47.550637 16932 raft_consensus.cc:686] T cb2776cf410841af944b68e2af842fc7 P 484d1f530fc44784a3be857749e2aec1 [term 1 LEADER]: Becoming Leader. State: Replica: 484d1f530fc44784a3be857749e2aec1, State: Running, Role: LEADER
I20260620 23:39:47.551174 16932 consensus_queue.cc:237] T cb2776cf410841af944b68e2af842fc7 P 484d1f530fc44784a3be857749e2aec1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e85b9450f44f4ba097cca79c9d37669c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42459 } } peers { permanent_uuid: "484d1f530fc44784a3be857749e2aec1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46215 } } peers { permanent_uuid: "dacd9f26586346c0ab4b9922f88308b3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45245 } }
I20260620 23:39:47.556320 16458 catalog_manager.cc:5495] T cb2776cf410841af944b68e2af842fc7 P 484d1f530fc44784a3be857749e2aec1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 484d1f530fc44784a3be857749e2aec1 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "484d1f530fc44784a3be857749e2aec1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e85b9450f44f4ba097cca79c9d37669c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "484d1f530fc44784a3be857749e2aec1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dacd9f26586346c0ab4b9922f88308b3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45245 } health_report { overall_health: UNKNOWN } } }
I20260620 23:39:47.576479 16920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:47.570727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60838 (local address 127.0.97.2:46215)
0620 23:39:47.570928 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 23:39:47.570979 (+ 51us) server_negotiation.cc:399] Waiting for connection header
0620 23:39:47.571011 (+ 32us) server_negotiation.cc:407] Connection header received
0620 23:39:47.571181 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:39:47.571185 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:39:47.571275 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:39:47.571567 (+ 292us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:39:47.571898 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:47.572688 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:47.574092 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:47.574586 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:47.574687 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:47.575311 (+ 624us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:39:47.575872 (+ 561us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:39:47.576010 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 23:39:47.576072 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 23:39:47.576140 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 23:39:47.585153 16862 raft_consensus.cc:1261] T cb2776cf410841af944b68e2af842fc7 P e85b9450f44f4ba097cca79c9d37669c [term 1 FOLLOWER]: Refusing update from remote peer 484d1f530fc44784a3be857749e2aec1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:39:47.585160 16599 raft_consensus.cc:1261] T cb2776cf410841af944b68e2af842fc7 P dacd9f26586346c0ab4b9922f88308b3 [term 1 FOLLOWER]: Refusing update from remote peer 484d1f530fc44784a3be857749e2aec1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:39:47.586143 16932 consensus_queue.cc:1035] T cb2776cf410841af944b68e2af842fc7 P 484d1f530fc44784a3be857749e2aec1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dacd9f26586346c0ab4b9922f88308b3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:39:47.586467 16930 consensus_queue.cc:1035] T cb2776cf410841af944b68e2af842fc7 P 484d1f530fc44784a3be857749e2aec1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e85b9450f44f4ba097cca79c9d37669c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:39:47.597572 16937 mvcc.cc:205] Tried to move back new op lower bound from 7299067033937981440 to 7299067033813823488. Current Snapshot: MvccSnapshot[applied={T|T < 7299067033937981440}]
I20260620 23:39:47.611078 16458 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56002:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:39:47.611367 16458 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56002:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:39:47.615047 16458 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b891c2846180438488602be42d467117: Sending DeleteTablet for 3 replicas of tablet cb2776cf410841af944b68e2af842fc7
I20260620 23:39:47.616329 16711 tablet_service.cc:1514] Processing DeleteTablet for tablet cb2776cf410841af944b68e2af842fc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:39:47 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60830
I20260620 23:39:47.616410 16842 tablet_service.cc:1514] Processing DeleteTablet for tablet cb2776cf410841af944b68e2af842fc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:39:47 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44062
I20260620 23:39:47.616621 16579 tablet_service.cc:1514] Processing DeleteTablet for tablet cb2776cf410841af944b68e2af842fc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:39:47 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34516
I20260620 23:39:47.618137 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 16496
I20260620 23:39:47.638859 16948 tablet_replica.cc:330] T cb2776cf410841af944b68e2af842fc7 P dacd9f26586346c0ab4b9922f88308b3: stopping tablet replica
I20260620 23:39:47.640879 16948 raft_consensus.cc:2229] T cb2776cf410841af944b68e2af842fc7 P dacd9f26586346c0ab4b9922f88308b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:39:47.643353 16922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:47.619427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34528 (local address 127.0.97.1:45245)
0620 23:39:47.619602 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 23:39:47.619609 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:39:47.619627 (+ 18us) server_negotiation.cc:407] Connection header received
0620 23:39:47.619690 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:39:47.619695 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:39:47.619811 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:39:47.619926 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:39:47.622155 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:47.623421 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:47.625894 (+ 2473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:47.626618 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:47.626749 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:47.635022 (+ 8273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:39:47.635044 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:39:47.635065 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:39:47.635125 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:39:47.639519 (+ 4394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:39:47.640711 (+ 1192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:39:47.640721 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:39:47.640724 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:39:47.640896 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:39:47.641365 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:39:47.641370 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:39:47.641373 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:39:47.641909 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:39:47.642245 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0620 23:39:47.642646 (+ 401us) server_negotiation.cc:299] Negotiation successful
0620 23:39:47.642899 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 23:39:47.644156 16639 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:39:47.644377 16946 tablet_replica.cc:330] T cb2776cf410841af944b68e2af842fc7 P 484d1f530fc44784a3be857749e2aec1: stopping tablet replica
I20260620 23:39:47.644881 16946 raft_consensus.cc:2229] T cb2776cf410841af944b68e2af842fc7 P 484d1f530fc44784a3be857749e2aec1 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:39:47.645709 16946 raft_consensus.cc:2258] T cb2776cf410841af944b68e2af842fc7 P 484d1f530fc44784a3be857749e2aec1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:39:47.648252 16946 ts_tablet_manager.cc:1905] T cb2776cf410841af944b68e2af842fc7 P 484d1f530fc44784a3be857749e2aec1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:39:47.651865 16947 tablet_replica.cc:330] T cb2776cf410841af944b68e2af842fc7 P e85b9450f44f4ba097cca79c9d37669c: stopping tablet replica
I20260620 23:39:47.652359 16947 raft_consensus.cc:2229] T cb2776cf410841af944b68e2af842fc7 P e85b9450f44f4ba097cca79c9d37669c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:39:47.652832 16947 raft_consensus.cc:2258] T cb2776cf410841af944b68e2af842fc7 P e85b9450f44f4ba097cca79c9d37669c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:39:47.656448 16946 ts_tablet_manager.cc:1918] T cb2776cf410841af944b68e2af842fc7 P 484d1f530fc44784a3be857749e2aec1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:39:47.656716 16946 log.cc:1196] T cb2776cf410841af944b68e2af842fc7 P 484d1f530fc44784a3be857749e2aec1: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-1/wal/wals/cb2776cf410841af944b68e2af842fc7
I20260620 23:39:47.657618 16946 ts_tablet_manager.cc:1939] T cb2776cf410841af944b68e2af842fc7 P 484d1f530fc44784a3be857749e2aec1: Deleting consensus metadata
I20260620 23:39:47.658895 16947 ts_tablet_manager.cc:1905] T cb2776cf410841af944b68e2af842fc7 P e85b9450f44f4ba097cca79c9d37669c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:39:47.660027 16445 catalog_manager.cc:4841] TS 484d1f530fc44784a3be857749e2aec1 (127.0.97.2:46215): tablet cb2776cf410841af944b68e2af842fc7 (table test-table [id=4288458361f345019fd7e6e672c591b5]) successfully deleted
I20260620 23:39:47.664243 16947 ts_tablet_manager.cc:1918] T cb2776cf410841af944b68e2af842fc7 P e85b9450f44f4ba097cca79c9d37669c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:39:47.664441 16947 log.cc:1196] T cb2776cf410841af944b68e2af842fc7 P e85b9450f44f4ba097cca79c9d37669c: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-2/wal/wals/cb2776cf410841af944b68e2af842fc7
I20260620 23:39:47.665066 16947 ts_tablet_manager.cc:1939] T cb2776cf410841af944b68e2af842fc7 P e85b9450f44f4ba097cca79c9d37669c: Deleting consensus metadata
I20260620 23:39:47.667038 16444 catalog_manager.cc:4841] TS e85b9450f44f4ba097cca79c9d37669c (127.0.97.3:42459): tablet cb2776cf410841af944b68e2af842fc7 (table test-table [id=4288458361f345019fd7e6e672c591b5]) successfully deleted
I20260620 23:39:47.752316 16948 raft_consensus.cc:2258] T cb2776cf410841af944b68e2af842fc7 P dacd9f26586346c0ab4b9922f88308b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:39:47.753077 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 16496
W20260620 23:39:47.771198 16444 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:45245 (error 108)
I20260620 23:39:47.771898 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 16648
I20260620 23:39:47.773415 16921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:47.772904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45245 (local address 127.0.0.1:34540)
0620 23:39:47.773111 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:47.773240 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260620 23:39:47.773702 16444 catalog_manager.cc:4584] TS dacd9f26586346c0ab4b9922f88308b3 (127.0.97.1:45245): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cb2776cf410841af944b68e2af842fc7: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45245: connect: Connection refused (error 111)
I20260620 23:39:47.792275 16920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:47.772767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60842 (local address 127.0.97.2:46215)
0620 23:39:47.773308 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0620 23:39:47.773314 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:39:47.773337 (+ 23us) server_negotiation.cc:407] Connection header received
0620 23:39:47.773404 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:39:47.773410 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:39:47.773492 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:39:47.773595 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:39:47.775051 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:47.776301 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:47.777602 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:47.778300 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:47.778433 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:47.784968 (+ 6535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:39:47.784987 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:39:47.785006 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:39:47.785069 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:39:47.789118 (+ 4049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:39:47.789982 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:39:47.789992 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:39:47.789996 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:39:47.790197 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:39:47.790589 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:39:47.790595 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:39:47.790598 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:39:47.791071 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:39:47.791437 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0620 23:39:47.791623 (+ 186us) server_negotiation.cc:299] Negotiation successful
0620 23:39:47.791857 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298}
I20260620 23:39:47.793081 16771 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:39:47.827682 16921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:47.827084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45245 (local address 127.0.0.1:34546)
0620 23:39:47.827391 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:47.827511 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171}
I20260620 23:39:47.828748 16921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:47.828393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45245 (local address 127.0.0.1:34548)
0620 23:39:47.828551 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:47.828615 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 23:39:47.889245 16921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:47.888716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45245 (local address 127.0.0.1:34552)
0620 23:39:47.888976 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:47.889068 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260620 23:39:47.890264 16921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:47.889935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45245 (local address 127.0.0.1:34566)
0620 23:39:47.890082 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:47.890158 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 23:39:47.909906 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 16648
I20260620 23:39:47.929677 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 16779
I20260620 23:39:47.943013 16919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:47.930469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44078 (local address 127.0.97.3:42459)
0620 23:39:47.930637 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 23:39:47.930641 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:39:47.930659 (+ 18us) server_negotiation.cc:407] Connection header received
0620 23:39:47.930745 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:39:47.930749 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:39:47.930833 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:39:47.930922 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:39:47.932278 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:47.933002 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:47.934213 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:39:47.934662 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:39:47.934765 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:39:47.937429 (+ 2664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:39:47.937445 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:39:47.937460 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:39:47.937493 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:39:47.940527 (+ 3034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:39:47.941253 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:39:47.941262 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:39:47.941264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:39:47.941409 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:39:47.941727 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:39:47.941731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:39:47.941735 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:39:47.942087 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:39:47.942324 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0620 23:39:47.942523 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 23:39:47.942701 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 23:39:47.943759 16902 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:39:47.970897 16921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:47.970416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45245 (local address 127.0.0.1:34574)
0620 23:39:47.970639 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:47.970737 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260620 23:39:47.972069 16921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:47.971673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45245 (local address 127.0.0.1:34590)
0620 23:39:47.971874 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:47.971966 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260620 23:39:48.036721 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 16779
I20260620 23:39:48.053562 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 16426
I20260620 23:39:48.054777 16489 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:39:48.170271 16921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:39:48.169488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45245 (local address 127.0.0.1:34596)
0620 23:39:48.169850 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0620 23:39:48.169966 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181}
I20260620 23:39:48.170400 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 16426
I20260620 23:40:08.192817 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37589
--webserver_interface=127.0.97.62
--webserver_port=35631
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37589
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:40:08.531888 16954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:40:08.532215 16954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:40:08.532382 16954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:40:08.539786 16954 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 23:40:08.539896 16954 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 23:40:08.539968 16954 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 23:40:08.540052 16954 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:40:08.540148 16954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:40:08.540269 16954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:40:08.540349 16954 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:40:08.540438 16954 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:40:08.550606 16954 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37589
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=35631
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:40:08.552111 16954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:40:08.553536 16954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:40:08.566918 16959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:40:08.566890 16962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:40:08.566957 16960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:40:08.567595 16954 server_base.cc:1034] running on GCE node
I20260620 23:40:08.568604 16954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:40:08.570290 16954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:40:08.571575 16954 hybrid_clock.cc:648] HybridClock initialized: now 1781998808571507 us; error 49 us; skew 500 ppm
I20260620 23:40:08.571980 16954 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:40:08 dist-test-slave-tcgx krb5kdc[15873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998808, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:40:08.576435 16954 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 23:40:08.578274 16954 webserver.cc:460] Webserver started at http://127.0.97.62:35631/ using document root <none> and password file <none>
I20260620 23:40:08.579080 16954 fs_manager.cc:359] Metadata directory not provided
I20260620 23:40:08.579178 16954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:40:08.584770 16954 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260620 23:40:08.587855 16969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:08.589478 16954 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:40:08.589705 16954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/master-0/wal
uuid: "b891c2846180438488602be42d467117"
format_stamp: "Formatted at 2026-06-20 23:39:22 on dist-test-slave-tcgx"
server_key: "e941f9af04805e6454aa55d5cd65b181"
server_key_iv: "214e658b56ac36088703a664b5f2dbd8"
server_key_version: "encryptionkey@0"
I20260620 23:40:08.590793 16954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:40:08.621127 16954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:40:08.626149 16954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:40:08.626518 16954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:40:08.647606 16954 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37589
I20260620 23:40:08.647717 17020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37589 every 8 connection(s)
I20260620 23:40:08.649376 16954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:40:08.657115 17021 sys_catalog.cc:263] Verifying existing consensus state
I20260620 23:40:08.658406 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 16954
I20260620 23:40:08.659552 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:45245
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:37589
--webserver_port=42213
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:40:08.662423 17021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b891c2846180438488602be42d467117: Bootstrap starting.
I20260620 23:40:08.686062 17021 log.cc:824] T 00000000000000000000000000000000 P b891c2846180438488602be42d467117: Log is configured to *not* fsync() on all Append() calls
I20260620 23:40:08.704083 17021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b891c2846180438488602be42d467117: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:40:08.704701 17021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b891c2846180438488602be42d467117: Bootstrap complete.
I20260620 23:40:08.714499 17021 raft_consensus.cc:348] T 00000000000000000000000000000000 P b891c2846180438488602be42d467117 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b891c2846180438488602be42d467117" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37589 } }
I20260620 23:40:08.715584 17021 raft_consensus.cc:729] T 00000000000000000000000000000000 P b891c2846180438488602be42d467117 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: b891c2846180438488602be42d467117, State: Initialized, Role: FOLLOWER
I20260620 23:40:08.716261 17021 consensus_queue.cc:260] T 00000000000000000000000000000000 P b891c2846180438488602be42d467117 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b891c2846180438488602be42d467117" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37589 } }
I20260620 23:40:08.716529 17021 raft_consensus.cc:388] T 00000000000000000000000000000000 P b891c2846180438488602be42d467117 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:40:08.716666 17021 raft_consensus.cc:482] T 00000000000000000000000000000000 P b891c2846180438488602be42d467117 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:40:08.716856 17021 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b891c2846180438488602be42d467117 [term 2 FOLLOWER]: Advancing to term 3
I20260620 23:40:08.721072 17021 raft_consensus.cc:504] T 00000000000000000000000000000000 P b891c2846180438488602be42d467117 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b891c2846180438488602be42d467117" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37589 } }
I20260620 23:40:08.721596 17021 leader_election.cc:302] T 00000000000000000000000000000000 P b891c2846180438488602be42d467117 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b891c2846180438488602be42d467117; no voters:
I20260620 23:40:08.722304 17021 leader_election.cc:288] T 00000000000000000000000000000000 P b891c2846180438488602be42d467117 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 23:40:08.722455 17025 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b891c2846180438488602be42d467117 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 23:40:08.723717 17025 raft_consensus.cc:686] T 00000000000000000000000000000000 P b891c2846180438488602be42d467117 [term 3 LEADER]: Becoming Leader. State: Replica: b891c2846180438488602be42d467117, State: Running, Role: LEADER
I20260620 23:40:08.724377 17025 consensus_queue.cc:237] T 00000000000000000000000000000000 P b891c2846180438488602be42d467117 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b891c2846180438488602be42d467117" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37589 } }
I20260620 23:40:08.724671 17021 sys_catalog.cc:564] T 00000000000000000000000000000000 P b891c2846180438488602be42d467117 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:40:08.727203 17027 sys_catalog.cc:455] T 00000000000000000000000000000000 P b891c2846180438488602be42d467117 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b891c2846180438488602be42d467117. Latest consensus state: current_term: 3 leader_uuid: "b891c2846180438488602be42d467117" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b891c2846180438488602be42d467117" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37589 } } }
I20260620 23:40:08.727628 17027 sys_catalog.cc:458] T 00000000000000000000000000000000 P b891c2846180438488602be42d467117 [sys.catalog]: This master's current role is: LEADER
I20260620 23:40:08.728199 17026 sys_catalog.cc:455] T 00000000000000000000000000000000 P b891c2846180438488602be42d467117 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "b891c2846180438488602be42d467117" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b891c2846180438488602be42d467117" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37589 } } }
I20260620 23:40:08.728515 17026 sys_catalog.cc:458] T 00000000000000000000000000000000 P b891c2846180438488602be42d467117 [sys.catalog]: This master's current role is: LEADER
I20260620 23:40:08.732667 17032 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:40:08.744184 17032 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:40:08.746450 17032 catalog_manager.cc:1259] Loaded cluster ID: 6bfd6b9c73a3459584b5ce669768df17
I20260620 23:40:08.746711 17032 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:40:08.752460 17032 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:40:08.754868 17032 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b891c2846180438488602be42d467117: Loaded TSK: 0
I20260620 23:40:08.755915 17032 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b891c2846180438488602be42d467117: Loaded TSK: 1
F20260620 23:40:08.757582 17032 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f902146b9ad google::LogMessage::Fail() at ??:0
@ 0x7f902146e88a google::LogMessage::SendToLog() at ??:0
@ 0x7f902146b4de google::LogMessage::Flush() at ??:0
@ 0x7f902146ca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f90251cf7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f90251be73e kudu::security::TokenSigner::ImportKeys() at ??:0
I20260620 23:40:08.774892 17043 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7f9036e9a1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f9036e99783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f9036f252a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f9036f250c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f9036f6f5e9 std::function<>::operator()() at ??:0
@ 0x7f9036e9dcdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f9036e9bc36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f9036f23610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f9036f23439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f90240b7345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f90240ba91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f90240ba799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f9024095eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f902736f6db start_thread at ??:0
@ 0x7f902030561f clone at ??:0
W20260620 23:40:09.054373 17023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:40:09.054733 17023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:40:09.054905 17023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:40:09.062397 17023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:40:09.062565 17023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:40:09.062752 17023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 23:40:09.072945 17023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:45245
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=42213
--tserver_master_addrs=127.0.97.62:37589
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:40:09.074352 17023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:40:09.075721 17023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:40:09.089159 17071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:40:09.089218 17068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:40:09.089193 17069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:40:09.089974 17023 server_base.cc:1034] running on GCE node
I20260620 23:40:09.090632 17023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:40:09.091671 17023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:40:09.092919 17023 hybrid_clock.cc:648] HybridClock initialized: now 1781998809092874 us; error 38 us; skew 500 ppm
I20260620 23:40:09.093374 17023 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:40:09 dist-test-slave-tcgx krb5kdc[15873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:40:09.096800 17023 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 23:40:09.098443 17023 webserver.cc:460] Webserver started at http://127.0.97.1:42213/ using document root <none> and password file <none>
I20260620 23:40:09.099251 17023 fs_manager.cc:359] Metadata directory not provided
I20260620 23:40:09.099406 17023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:40:09.105576 17023 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 23:40:09.108875 17078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:09.110509 17023 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:40:09.110716 17023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-0/wal
uuid: "dacd9f26586346c0ab4b9922f88308b3"
format_stamp: "Formatted at 2026-06-20 23:39:23 on dist-test-slave-tcgx"
server_key: "4ddd36bdf282d4bd73ad7efdc123b3f6"
server_key_iv: "a80e365783dc04e479250ba41387a93c"
server_key_version: "encryptionkey@0"
I20260620 23:40:09.111883 17023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:40:09.127275 17023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:40:09.131589 17023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:40:09.131886 17023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:40:09.132932 17023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:40:09.134843 17085 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260620 23:40:09.141799 17023 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 23:40:09.141952 17023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.008s user 0.001s sys 0.000s
I20260620 23:40:09.142102 17023 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 23:40:09.145328 17023 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 23:40:09.145457 17023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.001s sys 0.000s
I20260620 23:40:09.145802 17085 tablet_bootstrap.cc:492] T cb2776cf410841af944b68e2af842fc7 P dacd9f26586346c0ab4b9922f88308b3: Bootstrap starting.
I20260620 23:40:09.173741 17085 log.cc:824] T cb2776cf410841af944b68e2af842fc7 P dacd9f26586346c0ab4b9922f88308b3: Log is configured to *not* fsync() on all Append() calls
I20260620 23:40:09.186028 17023 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45245
I20260620 23:40:09.186079 17192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45245 every 8 connection(s)
I20260620 23:40:09.187497 17023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
I20260620 23:40:09.187759 17085 tablet_bootstrap.cc:492] T cb2776cf410841af944b68e2af842fc7 P dacd9f26586346c0ab4b9922f88308b3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:40:09.188431 17085 tablet_bootstrap.cc:492] T cb2776cf410841af944b68e2af842fc7 P dacd9f26586346c0ab4b9922f88308b3: Bootstrap complete.
I20260620 23:40:09.189589 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 17023
I20260620 23:40:09.190383 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:46215
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:37589
--webserver_port=33373
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:40:09.191406 17085 ts_tablet_manager.cc:1397] T cb2776cf410841af944b68e2af842fc7 P dacd9f26586346c0ab4b9922f88308b3: Time spent bootstrapping tablet: real 0.046s user 0.034s sys 0.009s
I20260620 23:40:09.197351 17195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:09.190702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37589 (local address 127.0.97.1:51503)
0620 23:40:09.195646 (+ 4944us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:09.195952 (+ 306us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4576,"thread_start_us":4340,"threads_started":1}
W20260620 23:40:09.198268 17193 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:37589 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:37589: Client connection negotiation failed: client connection to 127.0.97.62:37589: connect: Connection refused (error 111)
I20260620 23:40:09.199333 17195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:09.198914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37589 (local address 127.0.97.1:44587)
0620 23:40:09.199119 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:09.199202 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260620 23:40:09.200472 17195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:09.199912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37589 (local address 127.0.97.1:36645)
0620 23:40:09.200126 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:09.200243 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260620 23:40:09.200783 17193 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 23:40:09.199849 17085 raft_consensus.cc:348] T cb2776cf410841af944b68e2af842fc7 P dacd9f26586346c0ab4b9922f88308b3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e85b9450f44f4ba097cca79c9d37669c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42459 } } peers { permanent_uuid: "484d1f530fc44784a3be857749e2aec1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46215 } } peers { permanent_uuid: "dacd9f26586346c0ab4b9922f88308b3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45245 } }
I20260620 23:40:09.201135 17085 raft_consensus.cc:729] T cb2776cf410841af944b68e2af842fc7 P dacd9f26586346c0ab4b9922f88308b3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dacd9f26586346c0ab4b9922f88308b3, State: Initialized, Role: FOLLOWER
I20260620 23:40:09.201807 17085 consensus_queue.cc:260] T cb2776cf410841af944b68e2af842fc7 P dacd9f26586346c0ab4b9922f88308b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e85b9450f44f4ba097cca79c9d37669c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42459 } } peers { permanent_uuid: "484d1f530fc44784a3be857749e2aec1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46215 } } peers { permanent_uuid: "dacd9f26586346c0ab4b9922f88308b3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45245 } }
I20260620 23:40:09.203361 17085 ts_tablet_manager.cc:1428] T cb2776cf410841af944b68e2af842fc7 P dacd9f26586346c0ab4b9922f88308b3: Time spent starting tablet: real 0.012s user 0.007s sys 0.005s
I20260620 23:40:09.207379 17195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:09.203100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37589 (local address 127.0.97.1:59101)
0620 23:40:09.207153 (+ 4053us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:09.207237 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3938}
I20260620 23:40:09.208380 17195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:09.208074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37589 (local address 127.0.97.1:38951)
0620 23:40:09.208205 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:09.208268 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 23:40:09.542783 17196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:40:09.543176 17196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:40:09.543394 17196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:40:09.550871 17196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:40:09.551095 17196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:40:09.551307 17196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 23:40:09.561875 17196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:46215
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=33373
--tserver_master_addrs=127.0.97.62:37589
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:40:09.563335 17196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:40:09.564757 17196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:40:09.578809 17203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:40:09.579015 17202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:40:09.579413 17205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:40:09.579864 17196 server_base.cc:1034] running on GCE node
I20260620 23:40:09.580583 17196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:40:09.581713 17196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:40:09.583014 17196 hybrid_clock.cc:648] HybridClock initialized: now 1781998809582911 us; error 63 us; skew 500 ppm
I20260620 23:40:09.583352 17196 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:40:09 dist-test-slave-tcgx krb5kdc[15873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:40:09.586663 17196 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 23:40:09.588754 17196 webserver.cc:460] Webserver started at http://127.0.97.2:33373/ using document root <none> and password file <none>
I20260620 23:40:09.589522 17196 fs_manager.cc:359] Metadata directory not provided
I20260620 23:40:09.589689 17196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:40:09.597455 17196 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 23:40:09.601230 17212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:09.602818 17196 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:40:09.603047 17196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-1/wal
uuid: "484d1f530fc44784a3be857749e2aec1"
format_stamp: "Formatted at 2026-06-20 23:39:23 on dist-test-slave-tcgx"
server_key: "3f903f91fa429b32b3990ca10bc07ecc"
server_key_iv: "78277e1329cf21e3a17e33fad220cc3e"
server_key_version: "encryptionkey@0"
I20260620 23:40:09.604210 17196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:40:09.630218 17196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:40:09.635057 17196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:40:09.635325 17196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:40:09.636410 17196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:40:09.637959 17196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:40:09.638060 17196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:09.638159 17196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:40:09.638221 17196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:09.689868 17196 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46215
I20260620 23:40:09.689970 17324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46215 every 8 connection(s)
I20260620 23:40:09.691497 17196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
I20260620 23:40:09.693608 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 17196
I20260620 23:40:09.694311 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:42459
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:37589
--webserver_port=35389
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:40:09.697264 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:09.694904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37589 (local address 127.0.97.2:42347)
0620 23:40:09.695883 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:09.696204 (+ 321us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":356,"threads_started":1}
W20260620 23:40:09.698015 17325 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:37589 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:37589: Client connection negotiation failed: client connection to 127.0.97.62:37589: connect: Connection refused (error 111)
I20260620 23:40:09.699118 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:09.698682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37589 (local address 127.0.97.2:43451)
0620 23:40:09.698847 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:09.698922 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 23:40:09.700217 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:09.699760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37589 (local address 127.0.97.2:35577)
0620 23:40:09.699900 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:09.700054 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 23:40:09.700407 17325 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 23:40:10.037128 17328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:40:10.037441 17328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:40:10.037642 17328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:40:10.045056 17328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:40:10.045183 17328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:40:10.045378 17328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 23:40:10.055428 17328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:42459
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=35389
--tserver_master_addrs=127.0.97.62:37589
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:40:10.056782 17328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:40:10.058244 17328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:40:10.071702 17336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:40:10.071949 17333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:40:10.071965 17334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:40:10.073035 17328 server_base.cc:1034] running on GCE node
I20260620 23:40:10.073686 17328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:40:10.074709 17328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:40:10.075941 17328 hybrid_clock.cc:648] HybridClock initialized: now 1781998810075872 us; error 56 us; skew 500 ppm
I20260620 23:40:10.076344 17328 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:40:10 dist-test-slave-tcgx krb5kdc[15873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998810, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:40:10.080026 17328 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 23:40:10.081679 17328 webserver.cc:460] Webserver started at http://127.0.97.3:35389/ using document root <none> and password file <none>
I20260620 23:40:10.082427 17328 fs_manager.cc:359] Metadata directory not provided
I20260620 23:40:10.082564 17328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:40:10.088164 17328 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 23:40:10.091251 17343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:10.092743 17328 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 23:40:10.092947 17328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-2/wal
uuid: "e85b9450f44f4ba097cca79c9d37669c"
format_stamp: "Formatted at 2026-06-20 23:39:24 on dist-test-slave-tcgx"
server_key: "e1740287c5d986a49ac6fcf0d0cf1805"
server_key_iv: "fc3d60b0e3ed9d7ffe927cdac0327937"
server_key_version: "encryptionkey@0"
I20260620 23:40:10.093979 17328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:40:10.113226 17328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:40:10.118427 17328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:40:10.118722 17328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:40:10.119894 17328 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:40:10.121366 17328 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:40:10.121507 17328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:10.121642 17328 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:40:10.121729 17328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:10.156349 17328 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42459
I20260620 23:40:10.156409 17455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42459 every 8 connection(s)
I20260620 23:40:10.157981 17328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
I20260620 23:40:10.161175 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 17328
I20260620 23:40:10.164022 17458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:10.161631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37589 (local address 127.0.97.3:48193)
0620 23:40:10.162678 (+ 1047us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:10.162914 (+ 236us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":386,"threads_started":1}
W20260620 23:40:10.164822 17456 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:37589 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:37589: Client connection negotiation failed: client connection to 127.0.97.62:37589: connect: Connection refused (error 111)
I20260620 23:40:10.165728 17458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:10.165377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37589 (local address 127.0.97.3:41197)
0620 23:40:10.165556 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:10.165622 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260620 23:40:10.166636 17458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:10.166317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37589 (local address 127.0.97.3:56623)
0620 23:40:10.166448 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:10.166528 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 23:40:10.166903 17456 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 23:40:10.210258 17460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:10.209458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37589 (local address 127.0.97.1:60409)
0620 23:40:10.209938 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:10.210043 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":226,"threads_started":1}
I20260620 23:40:10.702095 17461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:10.701072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37589 (local address 127.0.97.2:35933)
0620 23:40:10.701756 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:10.701850 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":416,"threads_started":1}
I20260620 23:40:11.040951 17462 raft_consensus.cc:482] T cb2776cf410841af944b68e2af842fc7 P dacd9f26586346c0ab4b9922f88308b3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:40:11.041244 17462 raft_consensus.cc:504] T cb2776cf410841af944b68e2af842fc7 P dacd9f26586346c0ab4b9922f88308b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e85b9450f44f4ba097cca79c9d37669c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42459 } } peers { permanent_uuid: "484d1f530fc44784a3be857749e2aec1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46215 } } peers { permanent_uuid: "dacd9f26586346c0ab4b9922f88308b3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45245 } }
I20260620 23:40:11.042819 17462 leader_election.cc:288] T cb2776cf410841af944b68e2af842fc7 P dacd9f26586346c0ab4b9922f88308b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e85b9450f44f4ba097cca79c9d37669c (127.0.97.3:42459), 484d1f530fc44784a3be857749e2aec1 (127.0.97.2:46215)
Jun 20 23:40:11 dist-test-slave-tcgx krb5kdc[15873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
Jun 20 23:40:11 dist-test-slave-tcgx krb5kdc[15873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 23:40:11.080700 17466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:11.043516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47361 (local address 127.0.97.2:46215)
0620 23:40:11.055389 (+ 11873us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:11.055412 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:11.056674 (+ 1262us) server_negotiation.cc:407] Connection header received
0620 23:40:11.058301 (+ 1627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:11.058308 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:11.058638 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:11.058985 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:40:11.061156 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:11.062561 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:11.064400 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:11.065023 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:11.065228 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:11.073690 (+ 8462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:40:11.073699 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:40:11.073717 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:40:11.073758 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:40:11.076863 (+ 3105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:40:11.077735 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:40:11.077743 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:40:11.077748 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:40:11.077898 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:40:11.078347 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:40:11.078357 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:40:11.078359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:40:11.078891 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:40:11.079145 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:11.079918 (+ 773us) server_negotiation.cc:299] Negotiation successful
0620 23:40:11.080268 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11645,"thread_start_us":11453,"threads_started":1}
I20260620 23:40:11.080729 17465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:11.043385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51561 (local address 127.0.97.3:42459)
0620 23:40:11.056194 (+ 12809us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:11.056212 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:11.056266 (+ 54us) server_negotiation.cc:407] Connection header received
0620 23:40:11.058200 (+ 1934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:11.058209 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:11.058604 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:11.058918 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:40:11.061192 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:11.062594 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:11.064406 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:11.065028 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:11.065231 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:11.073795 (+ 8564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:40:11.073802 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:40:11.073815 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:40:11.073853 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:40:11.076859 (+ 3006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:40:11.077743 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:40:11.077749 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:40:11.077754 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:40:11.077908 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:40:11.078369 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:40:11.078374 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:40:11.078384 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:40:11.078888 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:40:11.079162 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:11.079926 (+ 764us) server_negotiation.cc:299] Negotiation successful
0620 23:40:11.080288 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12617,"thread_start_us":12485,"threads_started":1}
I20260620 23:40:11.080776 17463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:11.043121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42459 (local address 127.0.97.1:51561)
0620 23:40:11.055719 (+ 12598us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:11.055773 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 23:40:11.056525 (+ 752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:40:11.059111 (+ 2586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:40:11.059120 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:40:11.060059 (+ 939us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:40:11.060959 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:11.060975 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:11.062763 (+ 1788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:11.062769 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:11.064251 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:11.064262 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:11.065273 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:11.065280 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:11.065574 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:11.070369 (+ 4795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:40:11.070399 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:40:11.073517 (+ 3118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:40:11.077016 (+ 3499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:40:11.077025 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:40:11.077045 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:40:11.077593 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:40:11.078020 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:40:11.078027 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:40:11.078033 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:40:11.078250 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:40:11.079091 (+ 841us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:40:11.079101 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:40:11.079430 (+ 329us) client_negotiation.cc:769] Sending connection context
0620 23:40:11.079698 (+ 268us) client_negotiation.cc:240] Negotiation successful
0620 23:40:11.080296 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12418,"mutex_wait_us":284,"spinlock_wait_cycles":50944,"thread_start_us":12251,"threads_started":1}
I20260620 23:40:11.080928 17464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:11.043361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46215 (local address 127.0.97.1:47361)
0620 23:40:11.056522 (+ 13161us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:11.056542 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:40:11.056649 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:40:11.059108 (+ 2459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:40:11.059120 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:40:11.060040 (+ 920us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:40:11.060949 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:11.060970 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:11.062758 (+ 1788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:11.062763 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:11.064249 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:11.064262 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:11.065247 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:11.065253 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:11.065579 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:11.070369 (+ 4790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:40:11.070396 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:40:11.073322 (+ 2926us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:40:11.077016 (+ 3694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:40:11.077030 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:40:11.077045 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:40:11.077578 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:40:11.078010 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:40:11.078016 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:40:11.078029 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:40:11.078226 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:40:11.079090 (+ 864us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:40:11.079098 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:40:11.079430 (+ 332us) client_negotiation.cc:769] Sending connection context
0620 23:40:11.079701 (+ 271us) client_negotiation.cc:240] Negotiation successful
0620 23:40:11.080453 (+ 752us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13057,"mutex_wait_us":412,"spinlock_wait_cycles":44672,"thread_start_us":12847,"threads_started":1}
W20260620 23:40:11.084087 17080 leader_election.cc:341] T cb2776cf410841af944b68e2af842fc7 P dacd9f26586346c0ab4b9922f88308b3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e85b9450f44f4ba097cca79c9d37669c (127.0.97.3:42459): Not found: Tablet not found: cb2776cf410841af944b68e2af842fc7
W20260620 23:40:11.084461 17081 leader_election.cc:341] T cb2776cf410841af944b68e2af842fc7 P dacd9f26586346c0ab4b9922f88308b3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 484d1f530fc44784a3be857749e2aec1 (127.0.97.2:46215): Not found: Tablet not found: cb2776cf410841af944b68e2af842fc7
I20260620 23:40:11.084641 17081 leader_election.cc:302] T cb2776cf410841af944b68e2af842fc7 P dacd9f26586346c0ab4b9922f88308b3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dacd9f26586346c0ab4b9922f88308b3; no voters: 484d1f530fc44784a3be857749e2aec1, e85b9450f44f4ba097cca79c9d37669c
I20260620 23:40:11.085065 17462 raft_consensus.cc:2726] T cb2776cf410841af944b68e2af842fc7 P dacd9f26586346c0ab4b9922f88308b3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:40:11.168323 17467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:11.167483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37589 (local address 127.0.97.3:45101)
0620 23:40:11.167955 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:11.168085 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":236,"threads_started":1}
I20260620 23:40:11.211714 17463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:11.211217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37589 (local address 127.0.97.1:37681)
0620 23:40:11.211399 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:11.211563 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 23:40:11.703933 17468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:11.703208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37589 (local address 127.0.97.2:59115)
0620 23:40:11.703680 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:11.703769 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":245,"threads_started":1}
I20260620 23:40:12.170166 17469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:12.169307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37589 (local address 127.0.97.3:40727)
0620 23:40:12.169847 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:12.169933 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":311,"threads_started":1}
I20260620 23:40:12.213662 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:12.212741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37589 (local address 127.0.97.1:43081)
0620 23:40:12.213327 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:12.213420 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":268,"threads_started":1}
I20260620 23:40:12.617897 17471 raft_consensus.cc:482] T cb2776cf410841af944b68e2af842fc7 P dacd9f26586346c0ab4b9922f88308b3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:40:12.618227 17471 raft_consensus.cc:504] T cb2776cf410841af944b68e2af842fc7 P dacd9f26586346c0ab4b9922f88308b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e85b9450f44f4ba097cca79c9d37669c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42459 } } peers { permanent_uuid: "484d1f530fc44784a3be857749e2aec1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46215 } } peers { permanent_uuid: "dacd9f26586346c0ab4b9922f88308b3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45245 } }
I20260620 23:40:12.619025 17471 leader_election.cc:288] T cb2776cf410841af944b68e2af842fc7 P dacd9f26586346c0ab4b9922f88308b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e85b9450f44f4ba097cca79c9d37669c (127.0.97.3:42459), 484d1f530fc44784a3be857749e2aec1 (127.0.97.2:46215)
W20260620 23:40:12.620126 17080 leader_election.cc:341] T cb2776cf410841af944b68e2af842fc7 P dacd9f26586346c0ab4b9922f88308b3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e85b9450f44f4ba097cca79c9d37669c (127.0.97.3:42459): Not found: Tablet not found: cb2776cf410841af944b68e2af842fc7
W20260620 23:40:12.620338 17081 leader_election.cc:341] T cb2776cf410841af944b68e2af842fc7 P dacd9f26586346c0ab4b9922f88308b3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 484d1f530fc44784a3be857749e2aec1 (127.0.97.2:46215): Not found: Tablet not found: cb2776cf410841af944b68e2af842fc7
I20260620 23:40:12.620471 17081 leader_election.cc:302] T cb2776cf410841af944b68e2af842fc7 P dacd9f26586346c0ab4b9922f88308b3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dacd9f26586346c0ab4b9922f88308b3; no voters: 484d1f530fc44784a3be857749e2aec1, e85b9450f44f4ba097cca79c9d37669c
I20260620 23:40:12.620772 17471 raft_consensus.cc:2726] T cb2776cf410841af944b68e2af842fc7 P dacd9f26586346c0ab4b9922f88308b3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:40:12.705669 17472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:12.704903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37589 (local address 127.0.97.2:44229)
0620 23:40:12.705378 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:12.705468 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":260,"threads_started":1}
I20260620 23:40:13.171986 17473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:13.171263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37589 (local address 127.0.97.3:33113)
0620 23:40:13.171738 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:13.171822 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":218,"threads_started":1}
I20260620 23:40:13.215787 17474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:13.214807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37589 (local address 127.0.97.1:56915)
0620 23:40:13.215506 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:13.215603 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":371,"threads_started":1}
I20260620 23:40:13.707489 17475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:13.706659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37589 (local address 127.0.97.2:34259)
0620 23:40:13.707198 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:13.707293 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":220,"threads_started":1}
I20260620 23:40:14.173677 17476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:14.172928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37589 (local address 127.0.97.3:52187)
0620 23:40:14.173392 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:14.173483 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":231,"threads_started":1}
I20260620 23:40:14.217792 17477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:14.217004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37589 (local address 127.0.97.1:53807)
0620 23:40:14.217521 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:14.217598 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":289,"threads_started":1}
I20260620 23:40:14.709249 17478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:14.708498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37589 (local address 127.0.97.2:37077)
0620 23:40:14.708980 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:14.709089 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":251,"threads_started":1}
I20260620 23:40:15.101154 17479 raft_consensus.cc:482] T cb2776cf410841af944b68e2af842fc7 P dacd9f26586346c0ab4b9922f88308b3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:40:15.101354 17479 raft_consensus.cc:504] T cb2776cf410841af944b68e2af842fc7 P dacd9f26586346c0ab4b9922f88308b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e85b9450f44f4ba097cca79c9d37669c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42459 } } peers { permanent_uuid: "484d1f530fc44784a3be857749e2aec1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46215 } } peers { permanent_uuid: "dacd9f26586346c0ab4b9922f88308b3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45245 } }
I20260620 23:40:15.101958 17479 leader_election.cc:288] T cb2776cf410841af944b68e2af842fc7 P dacd9f26586346c0ab4b9922f88308b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e85b9450f44f4ba097cca79c9d37669c (127.0.97.3:42459), 484d1f530fc44784a3be857749e2aec1 (127.0.97.2:46215)
W20260620 23:40:15.103099 17080 leader_election.cc:341] T cb2776cf410841af944b68e2af842fc7 P dacd9f26586346c0ab4b9922f88308b3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e85b9450f44f4ba097cca79c9d37669c (127.0.97.3:42459): Not found: Tablet not found: cb2776cf410841af944b68e2af842fc7
W20260620 23:40:15.103392 17081 leader_election.cc:341] T cb2776cf410841af944b68e2af842fc7 P dacd9f26586346c0ab4b9922f88308b3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 484d1f530fc44784a3be857749e2aec1 (127.0.97.2:46215): Not found: Tablet not found: cb2776cf410841af944b68e2af842fc7
I20260620 23:40:15.103591 17081 leader_election.cc:302] T cb2776cf410841af944b68e2af842fc7 P dacd9f26586346c0ab4b9922f88308b3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dacd9f26586346c0ab4b9922f88308b3; no voters: 484d1f530fc44784a3be857749e2aec1, e85b9450f44f4ba097cca79c9d37669c
I20260620 23:40:15.103948 17479 raft_consensus.cc:2726] T cb2776cf410841af944b68e2af842fc7 P dacd9f26586346c0ab4b9922f88308b3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:40:15.175633 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:15.174777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37589 (local address 127.0.97.3:37095)
0620 23:40:15.175327 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:15.175416 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":313,"threads_started":1}
I20260620 23:40:15.220125 17481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:15.218887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37589 (local address 127.0.97.1:52377)
0620 23:40:15.219645 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:15.219804 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":317,"threads_started":1}
I20260620 23:40:15.711124 17482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:15.710205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37589 (local address 127.0.97.2:32997)
0620 23:40:15.710753 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:15.710848 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":277,"threads_started":1}
I20260620 23:40:16.177363 17483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:16.176586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37589 (local address 127.0.97.3:34049)
0620 23:40:16.177097 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:16.177181 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":263,"threads_started":1}
I20260620 23:40:16.221918 17484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:16.221167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37589 (local address 127.0.97.1:36497)
0620 23:40:16.221630 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:16.221714 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":236,"threads_started":1}
I20260620 23:40:16.713048 17485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:16.712062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37589 (local address 127.0.97.2:39807)
0620 23:40:16.712690 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:16.712802 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":227,"threads_started":1}
I20260620 23:40:17.155056 17486 raft_consensus.cc:482] T cb2776cf410841af944b68e2af842fc7 P dacd9f26586346c0ab4b9922f88308b3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:40:17.155289 17486 raft_consensus.cc:504] T cb2776cf410841af944b68e2af842fc7 P dacd9f26586346c0ab4b9922f88308b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e85b9450f44f4ba097cca79c9d37669c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42459 } } peers { permanent_uuid: "484d1f530fc44784a3be857749e2aec1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46215 } } peers { permanent_uuid: "dacd9f26586346c0ab4b9922f88308b3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45245 } }
I20260620 23:40:17.156020 17486 leader_election.cc:288] T cb2776cf410841af944b68e2af842fc7 P dacd9f26586346c0ab4b9922f88308b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e85b9450f44f4ba097cca79c9d37669c (127.0.97.3:42459), 484d1f530fc44784a3be857749e2aec1 (127.0.97.2:46215)
W20260620 23:40:17.157007 17080 leader_election.cc:341] T cb2776cf410841af944b68e2af842fc7 P dacd9f26586346c0ab4b9922f88308b3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e85b9450f44f4ba097cca79c9d37669c (127.0.97.3:42459): Not found: Tablet not found: cb2776cf410841af944b68e2af842fc7
W20260620 23:40:17.157217 17081 leader_election.cc:341] T cb2776cf410841af944b68e2af842fc7 P dacd9f26586346c0ab4b9922f88308b3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 484d1f530fc44784a3be857749e2aec1 (127.0.97.2:46215): Not found: Tablet not found: cb2776cf410841af944b68e2af842fc7
I20260620 23:40:17.157352 17081 leader_election.cc:302] T cb2776cf410841af944b68e2af842fc7 P dacd9f26586346c0ab4b9922f88308b3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dacd9f26586346c0ab4b9922f88308b3; no voters: 484d1f530fc44784a3be857749e2aec1, e85b9450f44f4ba097cca79c9d37669c
I20260620 23:40:17.157629 17486 raft_consensus.cc:2726] T cb2776cf410841af944b68e2af842fc7 P dacd9f26586346c0ab4b9922f88308b3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:40:17.179220 17487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:17.178348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37589 (local address 127.0.97.3:48903)
0620 23:40:17.178878 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:17.179024 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":246,"threads_started":1}
I20260620 23:40:17.223694 17488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:17.222883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37589 (local address 127.0.97.1:39643)
0620 23:40:17.223394 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:17.223483 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":296,"threads_started":1}
I20260620 23:40:17.715003 17489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:17.714194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37589 (local address 127.0.97.2:55405)
0620 23:40:17.714643 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:17.714750 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":227,"threads_started":1}
I20260620 23:40:18.180812 17490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:18.180122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37589 (local address 127.0.97.3:56851)
0620 23:40:18.180542 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:18.180624 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":214,"threads_started":1}
I20260620 23:40:18.225666 17491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:18.224956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37589 (local address 127.0.97.1:49285)
0620 23:40:18.225405 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:18.225486 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":248,"threads_started":1}
I20260620 23:40:18.716858 17492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:18.715985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37589 (local address 127.0.97.2:53003)
0620 23:40:18.716586 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:18.716696 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":347,"threads_started":1}
I20260620 23:40:19.182526 17493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:19.181795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37589 (local address 127.0.97.3:40893)
0620 23:40:19.182278 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:19.182368 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":239,"threads_started":1}
I20260620 23:40:19.227455 17494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:19.226743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37589 (local address 127.0.97.1:34841)
0620 23:40:19.227202 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:19.227282 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":261,"threads_started":1}
I20260620 23:40:19.718847 17495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:19.717870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37589 (local address 127.0.97.2:39309)
0620 23:40:19.718569 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:19.718661 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":447,"threads_started":1}
I20260620 23:40:20.184650 17496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:20.183748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37589 (local address 127.0.97.3:45603)
0620 23:40:20.184355 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:20.184459 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":235,"threads_started":1}
I20260620 23:40:20.229354 17497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:20.228576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37589 (local address 127.0.97.1:58615)
0620 23:40:20.229076 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:20.229175 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":259,"threads_started":1}
I20260620 23:40:20.720762 17498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:20.719992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37589 (local address 127.0.97.2:56689)
0620 23:40:20.720497 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:20.720588 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":244,"threads_started":1}
I20260620 23:40:20.905404 17499 raft_consensus.cc:482] T cb2776cf410841af944b68e2af842fc7 P dacd9f26586346c0ab4b9922f88308b3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:40:20.905671 17499 raft_consensus.cc:504] T cb2776cf410841af944b68e2af842fc7 P dacd9f26586346c0ab4b9922f88308b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e85b9450f44f4ba097cca79c9d37669c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42459 } } peers { permanent_uuid: "484d1f530fc44784a3be857749e2aec1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46215 } } peers { permanent_uuid: "dacd9f26586346c0ab4b9922f88308b3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45245 } }
I20260620 23:40:20.906440 17499 leader_election.cc:288] T cb2776cf410841af944b68e2af842fc7 P dacd9f26586346c0ab4b9922f88308b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e85b9450f44f4ba097cca79c9d37669c (127.0.97.3:42459), 484d1f530fc44784a3be857749e2aec1 (127.0.97.2:46215)
W20260620 23:40:20.907442 17080 leader_election.cc:341] T cb2776cf410841af944b68e2af842fc7 P dacd9f26586346c0ab4b9922f88308b3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e85b9450f44f4ba097cca79c9d37669c (127.0.97.3:42459): Not found: Tablet not found: cb2776cf410841af944b68e2af842fc7
W20260620 23:40:20.907709 17081 leader_election.cc:341] T cb2776cf410841af944b68e2af842fc7 P dacd9f26586346c0ab4b9922f88308b3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 484d1f530fc44784a3be857749e2aec1 (127.0.97.2:46215): Not found: Tablet not found: cb2776cf410841af944b68e2af842fc7
I20260620 23:40:20.907814 17081 leader_election.cc:302] T cb2776cf410841af944b68e2af842fc7 P dacd9f26586346c0ab4b9922f88308b3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dacd9f26586346c0ab4b9922f88308b3; no voters: 484d1f530fc44784a3be857749e2aec1, e85b9450f44f4ba097cca79c9d37669c
I20260620 23:40:20.908123 17499 raft_consensus.cc:2726] T cb2776cf410841af944b68e2af842fc7 P dacd9f26586346c0ab4b9922f88308b3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:40:21.186553 17500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:21.185764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37589 (local address 127.0.97.3:55877)
0620 23:40:21.186295 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:21.186390 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":249,"threads_started":1}
I20260620 23:40:21.235014 17501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:21.233639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37589 (local address 127.0.97.1:35119)
0620 23:40:21.234373 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:21.234581 (+ 208us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":344,"threads_started":1}
I20260620 23:40:21.722859 17502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:21.722031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37589 (local address 127.0.97.2:57541)
0620 23:40:21.722585 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:21.722678 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":288,"threads_started":1}
I20260620 23:40:22.188409 17503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:22.187622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37589 (local address 127.0.97.3:54631)
0620 23:40:22.188152 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:22.188235 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":291,"threads_started":1}
I20260620 23:40:22.237154 17504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:22.236387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37589 (local address 127.0.97.1:42085)
0620 23:40:22.236887 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:22.236974 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":274,"threads_started":1}
I20260620 23:40:22.724747 17505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:22.723967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37589 (local address 127.0.97.2:38483)
0620 23:40:22.724443 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:22.724535 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":233,"threads_started":1}
I20260620 23:40:23.190263 17506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:23.189430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37589 (local address 127.0.97.3:48009)
0620 23:40:23.189973 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:23.190088 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":278,"threads_started":1}
I20260620 23:40:23.239017 17507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:23.238138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37589 (local address 127.0.97.1:52863)
0620 23:40:23.238652 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:23.238756 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":282,"threads_started":1}
I20260620 23:40:23.726492 17508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:23.725650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37589 (local address 127.0.97.2:57237)
0620 23:40:23.726202 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:23.726291 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":298,"threads_started":1}
I20260620 23:40:24.192221 17509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:24.191291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37589 (local address 127.0.97.3:59849)
0620 23:40:24.191868 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:24.191983 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":219,"threads_started":1}
I20260620 23:40:24.241250 17510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:24.240387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37589 (local address 127.0.97.1:40623)
0620 23:40:24.240945 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:24.241064 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":320,"threads_started":1}
I20260620 23:40:24.728376 17511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:24.727656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37589 (local address 127.0.97.2:59187)
0620 23:40:24.728126 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:24.728218 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":234,"threads_started":1}
I20260620 23:40:25.163916 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 17023
I20260620 23:40:25.183529 17512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:25.165065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52816 (local address 127.0.97.1:45245)
0620 23:40:25.165697 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:25.165705 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:25.165732 (+ 27us) server_negotiation.cc:407] Connection header received
0620 23:40:25.165793 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:25.165800 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:25.166022 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:25.166196 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:40:25.167861 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:25.168863 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:25.170198 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:25.170768 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:25.170929 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:25.176975 (+ 6046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:40:25.176991 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:40:25.177009 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:40:25.177055 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:40:25.180163 (+ 3108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:40:25.181035 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:40:25.181046 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:40:25.181048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:40:25.181203 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:40:25.181654 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:40:25.181660 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:40:25.181668 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:40:25.182091 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:40:25.182464 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:25.182791 (+ 327us) server_negotiation.cc:299] Negotiation successful
0620 23:40:25.183143 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":308,"threads_started":1}
I20260620 23:40:25.184863 17188 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:40:25.194161 17514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:25.193369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37589 (local address 127.0.97.3:42905)
0620 23:40:25.193887 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:25.193988 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":277,"threads_started":1}
I20260620 23:40:25.277554 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 17023
I20260620 23:40:25.296218 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 17196
I20260620 23:40:25.315166 17516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:25.297050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47140 (local address 127.0.97.2:46215)
0620 23:40:25.297673 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:25.297678 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:25.297695 (+ 17us) server_negotiation.cc:407] Connection header received
0620 23:40:25.297758 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:25.297762 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:25.297820 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:25.297913 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:40:25.299471 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:25.300260 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:25.301467 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:25.301885 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:25.301976 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:25.309465 (+ 7489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:40:25.309471 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:40:25.309474 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:40:25.309519 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:40:25.312350 (+ 2831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:40:25.313118 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:40:25.313124 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:40:25.313132 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:40:25.313271 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:40:25.313650 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:40:25.313655 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:40:25.313658 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:40:25.313941 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:40:25.314220 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:25.314518 (+ 298us) server_negotiation.cc:299] Negotiation successful
0620 23:40:25.314749 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":372,"threads_started":1}
I20260620 23:40:25.315918 17320 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:40:25.419425 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 17196
I20260620 23:40:25.436885 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 17328
I20260620 23:40:25.450066 17518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:25.437659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45692 (local address 127.0.97.3:42459)
0620 23:40:25.438297 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:25.438302 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:25.438318 (+ 16us) server_negotiation.cc:407] Connection header received
0620 23:40:25.438373 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:25.438376 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:25.438445 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:25.438534 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:40:25.439936 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:25.440728 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:25.441906 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:25.442360 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:25.442464 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:25.444978 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:40:25.444983 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:40:25.444986 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:40:25.445019 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:40:25.447806 (+ 2787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:40:25.448414 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:40:25.448417 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:40:25.448420 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:40:25.448543 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:40:25.448840 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:40:25.448843 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:40:25.448845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:40:25.449082 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:40:25.449292 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:25.449492 (+ 200us) server_negotiation.cc:299] Negotiation successful
0620 23:40:25.449651 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":366,"threads_started":1}
I20260620 23:40:25.450767 17451 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:40:25.538041 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 17328
2026-06-20T23:40:25Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64666 ms)
[----------] 30 tests from SecurityITest (299989 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:40:25 dist-test-slave-tcgx krb5kdc[17522](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:40:25 dist-test-slave-tcgx krb5kdc[17522](info): set up 2 sockets
Jun 20 23:40:25 dist-test-slave-tcgx krb5kdc[17522](info): commencing operation
krb5kdc: starting...
W20260620 23:40:26.663024 388 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/krb5kdc/test-user.keytab.
Jun 20 23:40:26 dist-test-slave-tcgx krb5kdc[17522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:40:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:40:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 23:40:26.918366 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44793
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44793
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:40:27.254065 17535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:40:27.254387 17535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:40:27.254545 17535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:40:27.262085 17535 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:40:27.262214 17535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:40:27.262328 17535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:40:27.262425 17535 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:40:27.262512 17535 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:40:27.272761 17535 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44793
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:40:27.274142 17535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:40:27.275560 17535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:40:27.287951 17540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:40:27.287990 17541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:40:27.288024 17543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:40:27.288641 17535 server_base.cc:1034] running on GCE node
I20260620 23:40:27.289537 17535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:40:27.291144 17535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:40:27.292364 17535 hybrid_clock.cc:648] HybridClock initialized: now 1781998827292302 us; error 62 us; skew 500 ppm
I20260620 23:40:27.292734 17535 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:40:27 dist-test-slave-tcgx krb5kdc[17522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998827, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:40:27.295949 17535 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 23:40:27.297432 17535 webserver.cc:460] Webserver started at http://127.0.97.62:34935/ using document root <none> and password file <none>
I20260620 23:40:27.298131 17535 fs_manager.cc:359] Metadata directory not provided
I20260620 23:40:27.298230 17535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:40:27.298512 17535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:40:27.301608 17535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/minicluster-data/master-0/data/instance:
uuid: "3303509587744568af792ad41b20da1e"
format_stamp: "Formatted at 2026-06-20 23:40:27 on dist-test-slave-tcgx"
server_key: "4ea5a40c5c745ffae9c3dde6aa37ffa0"
server_key_iv: "48c5c8b92cb865b1d1c205d136ffbec3"
server_key_version: "encryptionkey@0"
I20260620 23:40:27.302346 17535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/minicluster-data/master-0/wal/instance:
uuid: "3303509587744568af792ad41b20da1e"
format_stamp: "Formatted at 2026-06-20 23:40:27 on dist-test-slave-tcgx"
server_key: "4ea5a40c5c745ffae9c3dde6aa37ffa0"
server_key_iv: "48c5c8b92cb865b1d1c205d136ffbec3"
server_key_version: "encryptionkey@0"
I20260620 23:40:27.307857 17535 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 23:40:27.311480 17550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:27.312887 17535 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 23:40:27.313040 17535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/minicluster-data/master-0/wal
uuid: "3303509587744568af792ad41b20da1e"
format_stamp: "Formatted at 2026-06-20 23:40:27 on dist-test-slave-tcgx"
server_key: "4ea5a40c5c745ffae9c3dde6aa37ffa0"
server_key_iv: "48c5c8b92cb865b1d1c205d136ffbec3"
server_key_version: "encryptionkey@0"
I20260620 23:40:27.313217 17535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:40:27.346727 17535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:40:27.351091 17535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:40:27.351347 17535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:40:27.370185 17535 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44793
I20260620 23:40:27.370204 17601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44793 every 8 connection(s)
I20260620 23:40:27.371699 17535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:40:27.372633 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 17535
I20260620 23:40:27.372851 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/minicluster-data/master-0/wal/instance
I20260620 23:40:27.373508 388 external_mini_cluster.cc:1428] Setting key 648f8e26765e75d0c3e9f7cc801dd58a
I20260620 23:40:27.376212 17602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 23:40:27 dist-test-slave-tcgx krb5kdc[17522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:40:27.390143 17602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3303509587744568af792ad41b20da1e: Bootstrap starting.
I20260620 23:40:27.393642 17602 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3303509587744568af792ad41b20da1e: Neither blocks nor log segments found. Creating new log.
I20260620 23:40:27.394716 17602 log.cc:824] T 00000000000000000000000000000000 P 3303509587744568af792ad41b20da1e: Log is configured to *not* fsync() on all Append() calls
I20260620 23:40:27.397720 17605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:27.375552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38738 (local address 127.0.97.62:44793)
0620 23:40:27.376400 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:27.376412 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:27.376452 (+ 40us) server_negotiation.cc:407] Connection header received
0620 23:40:27.378146 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:27.378168 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:27.378573 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:27.378925 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:40:27.380779 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:27.382188 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:27.384213 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:27.384932 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:27.385069 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:27.390149 (+ 5080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:40:27.390160 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:40:27.390177 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:40:27.390227 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:40:27.393384 (+ 3157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:40:27.394212 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:40:27.394220 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:40:27.394227 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:40:27.394430 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:40:27.394871 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:40:27.394878 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:40:27.394881 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:40:27.395656 (+ 775us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:40:27.396012 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:27.396314 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 23:40:27.396726 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":358,"threads_started":1}
I20260620 23:40:27.398097 17602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3303509587744568af792ad41b20da1e: No bootstrap required, opened a new log
I20260620 23:40:27.406526 17602 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3303509587744568af792ad41b20da1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3303509587744568af792ad41b20da1e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44793 } }
I20260620 23:40:27.406919 17602 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3303509587744568af792ad41b20da1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:40:27.407064 17602 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3303509587744568af792ad41b20da1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3303509587744568af792ad41b20da1e, State: Initialized, Role: FOLLOWER
I20260620 23:40:27.407605 17602 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3303509587744568af792ad41b20da1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3303509587744568af792ad41b20da1e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44793 } }
I20260620 23:40:27.407812 17602 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3303509587744568af792ad41b20da1e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:40:27.407912 17602 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3303509587744568af792ad41b20da1e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:40:27.408041 17602 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3303509587744568af792ad41b20da1e [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:40:27.410178 17602 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3303509587744568af792ad41b20da1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3303509587744568af792ad41b20da1e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44793 } }
I20260620 23:40:27.410663 17602 leader_election.cc:302] T 00000000000000000000000000000000 P 3303509587744568af792ad41b20da1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3303509587744568af792ad41b20da1e; no voters:
I20260620 23:40:27.411377 17602 leader_election.cc:288] T 00000000000000000000000000000000 P 3303509587744568af792ad41b20da1e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:40:27.411530 17607 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3303509587744568af792ad41b20da1e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:40:27.412284 17607 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3303509587744568af792ad41b20da1e [term 1 LEADER]: Becoming Leader. State: Replica: 3303509587744568af792ad41b20da1e, State: Running, Role: LEADER
I20260620 23:40:27.412889 17607 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3303509587744568af792ad41b20da1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3303509587744568af792ad41b20da1e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44793 } }
I20260620 23:40:27.413642 17602 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3303509587744568af792ad41b20da1e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:40:27.416844 17608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3303509587744568af792ad41b20da1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3303509587744568af792ad41b20da1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3303509587744568af792ad41b20da1e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44793 } } }
I20260620 23:40:27.417368 17608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3303509587744568af792ad41b20da1e [sys.catalog]: This master's current role is: LEADER
I20260620 23:40:27.418165 17616 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:40:27.418165 17609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3303509587744568af792ad41b20da1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3303509587744568af792ad41b20da1e. Latest consensus state: current_term: 1 leader_uuid: "3303509587744568af792ad41b20da1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3303509587744568af792ad41b20da1e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44793 } } }
I20260620 23:40:27.418553 17609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3303509587744568af792ad41b20da1e [sys.catalog]: This master's current role is: LEADER
I20260620 23:40:27.422891 17616 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:40:27.432663 17616 catalog_manager.cc:1347] Generated new cluster ID: 9378fa5ae6084463977a44825b3149a9
I20260620 23:40:27.432789 17616 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:40:27.450834 17616 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:40:27.452212 17616 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:40:27.462153 17616 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3303509587744568af792ad41b20da1e: Generated new TSK 0
I20260620 23:40:27.463150 17616 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 23:40:27.531196 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44793
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:40:27.879335 17628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:40:27.879635 17628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:40:27.879789 17628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:40:27.887319 17628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:40:27.887439 17628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:40:27.887630 17628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 23:40:27.897926 17628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44793
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:40:27.899485 17628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:40:27.900866 17628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:40:27.916332 17634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:40:27.916419 17633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:40:27.916347 17636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:40:27.917300 17628 server_base.cc:1034] running on GCE node
I20260620 23:40:27.918061 17628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:40:27.919267 17628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:40:27.920485 17628 hybrid_clock.cc:648] HybridClock initialized: now 1781998827920412 us; error 60 us; skew 500 ppm
I20260620 23:40:27.920838 17628 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:40:27 dist-test-slave-tcgx krb5kdc[17522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998827, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:40:27.924177 17628 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 23:40:27.925642 17628 webserver.cc:460] Webserver started at http://127.0.97.1:44995/ using document root <none> and password file <none>
I20260620 23:40:27.926343 17628 fs_manager.cc:359] Metadata directory not provided
I20260620 23:40:27.926471 17628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:40:27.926792 17628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:40:27.929912 17628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/minicluster-data/ts-0/data/instance:
uuid: "3e7374a72999460ebbd606e772206d25"
format_stamp: "Formatted at 2026-06-20 23:40:27 on dist-test-slave-tcgx"
server_key: "ed61b22777608c61b466abf00eaf0936"
server_key_iv: "2235f46a837cae48f3fc99ff3b0f4772"
server_key_version: "encryptionkey@0"
I20260620 23:40:27.930761 17628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/minicluster-data/ts-0/wal/instance:
uuid: "3e7374a72999460ebbd606e772206d25"
format_stamp: "Formatted at 2026-06-20 23:40:27 on dist-test-slave-tcgx"
server_key: "ed61b22777608c61b466abf00eaf0936"
server_key_iv: "2235f46a837cae48f3fc99ff3b0f4772"
server_key_version: "encryptionkey@0"
I20260620 23:40:27.936154 17628 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 23:40:27.939754 17643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:27.941140 17628 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260620 23:40:27.941337 17628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/minicluster-data/ts-0/wal
uuid: "3e7374a72999460ebbd606e772206d25"
format_stamp: "Formatted at 2026-06-20 23:40:27 on dist-test-slave-tcgx"
server_key: "ed61b22777608c61b466abf00eaf0936"
server_key_iv: "2235f46a837cae48f3fc99ff3b0f4772"
server_key_version: "encryptionkey@0"
I20260620 23:40:27.941550 17628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:40:27.957899 17628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:40:27.962628 17628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:40:27.962926 17628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:40:27.963968 17628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:40:27.965360 17628 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:40:27.965459 17628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:27.965600 17628 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:40:27.965665 17628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:27.999233 17628 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38123
I20260620 23:40:27.999298 17755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38123 every 8 connection(s)
I20260620 23:40:28.000619 17628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
I20260620 23:40:28.009941 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 17628
I20260620 23:40:28.010166 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/minicluster-data/ts-0/wal/instance
I20260620 23:40:28.010809 388 external_mini_cluster.cc:1428] Setting key c74b980d5d4aa64b9e4c81da2485231c
Jun 20 23:40:28 dist-test-slave-tcgx krb5kdc[17522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998827, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:40:28.022478 17758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:28.003292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36649 (local address 127.0.97.62:44793)
0620 23:40:28.003746 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:28.003750 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:28.004999 (+ 1249us) server_negotiation.cc:407] Connection header received
0620 23:40:28.006930 (+ 1931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:28.006968 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:28.007065 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:28.007187 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:40:28.008872 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:28.009596 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:28.011412 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:28.011857 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:28.012015 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:28.016395 (+ 4380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:40:28.016400 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:40:28.016404 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:40:28.016442 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:40:28.019286 (+ 2844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:40:28.020093 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:40:28.020098 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:40:28.020100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:40:28.020227 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:40:28.020682 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:40:28.020688 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:40:28.020691 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:40:28.020968 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:40:28.021131 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:28.021879 (+ 748us) server_negotiation.cc:299] Negotiation successful
0620 23:40:28.022114 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":226,"threads_started":1}
I20260620 23:40:28.023571 17759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:28.003933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44793 (local address 127.0.97.1:36649)
0620 23:40:28.004821 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:28.004867 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 23:40:28.006619 (+ 1752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:40:28.007487 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:40:28.007496 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:40:28.008071 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:40:28.008680 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:28.008698 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:28.009778 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:28.009782 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:28.011253 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:28.011267 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:28.012098 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:28.012104 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:28.012339 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:28.013162 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:40:28.013195 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:40:28.016116 (+ 2921us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:40:28.019458 (+ 3342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:40:28.019467 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:40:28.019488 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:40:28.019955 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:40:28.020343 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:40:28.020347 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:40:28.020351 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:40:28.020568 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:40:28.021095 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:40:28.021106 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:40:28.021494 (+ 388us) client_negotiation.cc:769] Sending connection context
0620 23:40:28.021780 (+ 286us) client_negotiation.cc:240] Negotiation successful
0620 23:40:28.022371 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":439,"threads_started":1}
I20260620 23:40:28.025308 17756 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44793
I20260620 23:40:28.025619 17756 heartbeater.cc:464] Registering TS with master...
I20260620 23:40:28.026373 17756 heartbeater.cc:511] Master 127.0.97.62:44793 requested a full tablet report, sending...
I20260620 23:40:28.028591 17567 ts_manager.cc:194] Registered new tserver with Master: 3e7374a72999460ebbd606e772206d25 (127.0.97.1:38123)
I20260620 23:40:28.030042 17567 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:36649
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 23:40:28.066994 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44793
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:40:28.411842 17762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:40:28.412160 17762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:40:28.412259 17762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:40:28.419977 17762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:40:28.420125 17762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:40:28.420284 17762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 23:40:28.430470 17762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44793
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:40:28.432134 17762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:40:28.433485 17762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:40:28.447294 17767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:40:28.447374 17768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:40:28.447340 17770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:40:28.448364 17762 server_base.cc:1034] running on GCE node
I20260620 23:40:28.449028 17762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:40:28.450109 17762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:40:28.451414 17762 hybrid_clock.cc:648] HybridClock initialized: now 1781998828451340 us; error 58 us; skew 500 ppm
I20260620 23:40:28.451772 17762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:40:28 dist-test-slave-tcgx krb5kdc[17522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998828, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:40:28.455081 17762 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 23:40:28.456676 17762 webserver.cc:460] Webserver started at http://127.0.97.2:33391/ using document root <none> and password file <none>
I20260620 23:40:28.457422 17762 fs_manager.cc:359] Metadata directory not provided
I20260620 23:40:28.457553 17762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:40:28.457881 17762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:40:28.461159 17762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/minicluster-data/ts-1/data/instance:
uuid: "aa54fe804793430289a9e7b0163f24ec"
format_stamp: "Formatted at 2026-06-20 23:40:28 on dist-test-slave-tcgx"
server_key: "2bacf4d15bc3a5629e17f2c914202db4"
server_key_iv: "5bf94d783573da5f8bb0c1f29c90671f"
server_key_version: "encryptionkey@0"
I20260620 23:40:28.462050 17762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/minicluster-data/ts-1/wal/instance:
uuid: "aa54fe804793430289a9e7b0163f24ec"
format_stamp: "Formatted at 2026-06-20 23:40:28 on dist-test-slave-tcgx"
server_key: "2bacf4d15bc3a5629e17f2c914202db4"
server_key_iv: "5bf94d783573da5f8bb0c1f29c90671f"
server_key_version: "encryptionkey@0"
I20260620 23:40:28.467734 17762 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 23:40:28.471561 17777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:28.473155 17762 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 23:40:28.473353 17762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/minicluster-data/ts-1/wal
uuid: "aa54fe804793430289a9e7b0163f24ec"
format_stamp: "Formatted at 2026-06-20 23:40:28 on dist-test-slave-tcgx"
server_key: "2bacf4d15bc3a5629e17f2c914202db4"
server_key_iv: "5bf94d783573da5f8bb0c1f29c90671f"
server_key_version: "encryptionkey@0"
I20260620 23:40:28.473585 17762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:40:28.495363 17762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:40:28.500191 17762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:40:28.500526 17762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:40:28.501653 17762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:40:28.503127 17762 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:40:28.503206 17762 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:28.503387 17762 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:40:28.503443 17762 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:28.537791 17762 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45495
I20260620 23:40:28.537835 17889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45495 every 8 connection(s)
I20260620 23:40:28.539186 17762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
I20260620 23:40:28.542079 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 17762
I20260620 23:40:28.542315 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/minicluster-data/ts-1/wal/instance
I20260620 23:40:28.542831 388 external_mini_cluster.cc:1428] Setting key 0186defb71e98f48b43dd8e33e0a079e
Jun 20 23:40:28 dist-test-slave-tcgx krb5kdc[17522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998828, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:40:28.562498 17892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:28.541971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57295 (local address 127.0.97.62:44793)
0620 23:40:28.542460 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:28.542467 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:28.543635 (+ 1168us) server_negotiation.cc:407] Connection header received
0620 23:40:28.545678 (+ 2043us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:28.545686 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:28.545783 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:28.545909 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:40:28.547696 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:28.548857 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:28.550477 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:28.551256 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:28.551408 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:28.555152 (+ 3744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:40:28.555161 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:40:28.555169 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:40:28.555219 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:40:28.558835 (+ 3616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:40:28.559713 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:40:28.559721 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:40:28.559728 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:40:28.559899 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:40:28.560361 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:40:28.560369 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:40:28.560375 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:40:28.560775 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:40:28.560937 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:28.561732 (+ 795us) server_negotiation.cc:299] Negotiation successful
0620 23:40:28.561996 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":222,"threads_started":1}
I20260620 23:40:28.563365 17893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:28.542554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44793 (local address 127.0.97.2:57295)
0620 23:40:28.543446 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:28.543494 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 23:40:28.545383 (+ 1889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:40:28.546170 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:40:28.546179 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:40:28.546832 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:40:28.547508 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:28.547526 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:28.549016 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:28.549020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:28.550328 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:28.550340 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:28.551520 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:28.551526 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:28.551801 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:28.552538 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:40:28.552567 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:40:28.554854 (+ 2287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:40:28.559069 (+ 4215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:40:28.559076 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:40:28.559092 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:40:28.559568 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:40:28.560030 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:40:28.560034 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:40:28.560036 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:40:28.560234 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:40:28.560898 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:40:28.560904 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:40:28.561244 (+ 340us) client_negotiation.cc:769] Sending connection context
0620 23:40:28.561584 (+ 340us) client_negotiation.cc:240] Negotiation successful
0620 23:40:28.562023 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":380,"threads_started":1}
I20260620 23:40:28.564875 17890 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44793
I20260620 23:40:28.565201 17890 heartbeater.cc:464] Registering TS with master...
I20260620 23:40:28.565961 17890 heartbeater.cc:511] Master 127.0.97.62:44793 requested a full tablet report, sending...
I20260620 23:40:28.567461 17567 ts_manager.cc:194] Registered new tserver with Master: aa54fe804793430289a9e7b0163f24ec (127.0.97.2:45495)
I20260620 23:40:28.568300 17567 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:57295
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 23:40:28.602181 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44793
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:40:28.940164 17896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:40:28.940483 17896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:40:28.940582 17896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:40:28.948027 17896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:40:28.948154 17896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:40:28.948280 17896 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 23:40:28.958012 17896 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44793
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:40:28.959326 17896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:40:28.960623 17896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:40:28.973837 17901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:40:28.973856 17904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:40:28.973773 17902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:40:28.974658 17896 server_base.cc:1034] running on GCE node
I20260620 23:40:28.975327 17896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:40:28.976302 17896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:40:28.977531 17896 hybrid_clock.cc:648] HybridClock initialized: now 1781998828977469 us; error 54 us; skew 500 ppm
I20260620 23:40:28.977890 17896 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:40:28 dist-test-slave-tcgx krb5kdc[17522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998828, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:40:28.981317 17896 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 23:40:28.982760 17896 webserver.cc:460] Webserver started at http://127.0.97.3:41879/ using document root <none> and password file <none>
I20260620 23:40:28.983553 17896 fs_manager.cc:359] Metadata directory not provided
I20260620 23:40:28.983696 17896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:40:28.984015 17896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:40:28.987064 17896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/minicluster-data/ts-2/data/instance:
uuid: "db19a538c9b54628858e8def3f84ed34"
format_stamp: "Formatted at 2026-06-20 23:40:28 on dist-test-slave-tcgx"
server_key: "7aa30a30af5af0b1fa63a673b668ff17"
server_key_iv: "60f0a3dc0aadc2a4d002178577d869a0"
server_key_version: "encryptionkey@0"
I20260620 23:40:28.987893 17896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/minicluster-data/ts-2/wal/instance:
uuid: "db19a538c9b54628858e8def3f84ed34"
format_stamp: "Formatted at 2026-06-20 23:40:28 on dist-test-slave-tcgx"
server_key: "7aa30a30af5af0b1fa63a673b668ff17"
server_key_iv: "60f0a3dc0aadc2a4d002178577d869a0"
server_key_version: "encryptionkey@0"
I20260620 23:40:28.993273 17896 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260620 23:40:28.997066 17911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:28.998566 17896 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:40:28.998795 17896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/minicluster-data/ts-2/wal
uuid: "db19a538c9b54628858e8def3f84ed34"
format_stamp: "Formatted at 2026-06-20 23:40:28 on dist-test-slave-tcgx"
server_key: "7aa30a30af5af0b1fa63a673b668ff17"
server_key_iv: "60f0a3dc0aadc2a4d002178577d869a0"
server_key_version: "encryptionkey@0"
I20260620 23:40:28.999073 17896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:40:29.011765 17896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:40:29.016361 17896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:40:29.016660 17896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:40:29.017705 17896 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:40:29.019263 17896 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:40:29.019371 17896 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:29.019519 17896 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:40:29.019593 17896 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:29.033177 17756 heartbeater.cc:503] Master 127.0.97.62:44793 was elected leader, sending a full tablet report...
I20260620 23:40:29.052242 17896 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:32797
I20260620 23:40:29.052290 18023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:32797 every 8 connection(s)
I20260620 23:40:29.053531 17896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
I20260620 23:40:29.057933 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 17896
I20260620 23:40:29.058125 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/minicluster-data/ts-2/wal/instance
I20260620 23:40:29.058686 388 external_mini_cluster.cc:1428] Setting key 5089201a8570da9bd0498c599c42d53d
Jun 20 23:40:29 dist-test-slave-tcgx krb5kdc[17522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998828, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:40:29.073747 17892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:29.056158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50625 (local address 127.0.97.62:44793)
0620 23:40:29.056337 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:29.056342 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:29.057569 (+ 1227us) server_negotiation.cc:407] Connection header received
0620 23:40:29.059914 (+ 2345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:29.059919 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:29.060031 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:29.060143 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:40:29.062023 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:29.062827 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:29.064387 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:29.064766 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:29.064868 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:29.068219 (+ 3351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:40:29.068224 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:40:29.068227 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:40:29.068257 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:40:29.070748 (+ 2491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:40:29.071473 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:40:29.071478 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:40:29.071480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:40:29.071599 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:40:29.071955 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:40:29.071958 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:40:29.071960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:40:29.072213 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:40:29.072374 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:29.073197 (+ 823us) server_negotiation.cc:299] Negotiation successful
0620 23:40:29.073442 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 23:40:29.074819 18026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:29.056624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44793 (local address 127.0.97.3:50625)
0620 23:40:29.057352 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:29.057416 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 23:40:29.059660 (+ 2244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:40:29.060437 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:40:29.060451 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:40:29.061126 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:40:29.061860 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:29.061887 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:29.062999 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:29.063003 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:29.064265 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:29.064277 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:29.064951 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:29.064956 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:29.065144 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:29.065833 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:40:29.065857 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:40:29.068039 (+ 2182us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:40:29.070879 (+ 2840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:40:29.070886 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:40:29.070898 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:40:29.071372 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:40:29.071689 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:40:29.071694 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:40:29.071696 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:40:29.071863 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:40:29.072356 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:40:29.072361 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:40:29.072792 (+ 431us) client_negotiation.cc:769] Sending connection context
0620 23:40:29.073104 (+ 312us) client_negotiation.cc:240] Negotiation successful
0620 23:40:29.073644 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":282,"threads_started":1}
I20260620 23:40:29.076195 18024 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44793
I20260620 23:40:29.076489 18024 heartbeater.cc:464] Registering TS with master...
I20260620 23:40:29.077164 18024 heartbeater.cc:511] Master 127.0.97.62:44793 requested a full tablet report, sending...
I20260620 23:40:29.078598 17567 ts_manager.cc:194] Registered new tserver with Master: db19a538c9b54628858e8def3f84ed34 (127.0.97.3:32797)
I20260620 23:40:29.079443 17567 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:50625
I20260620 23:40:29.086371 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:40:29.103168 17892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:29.090740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38748 (local address 127.0.97.62:44793)
0620 23:40:29.090921 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:29.090926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:29.091395 (+ 469us) server_negotiation.cc:407] Connection header received
0620 23:40:29.091530 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:29.091534 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:29.091599 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:29.091683 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:40:29.093040 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:29.093777 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:29.095153 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:29.095537 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:29.095630 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:29.098012 (+ 2382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:40:29.098018 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:40:29.098021 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:40:29.098057 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:40:29.100735 (+ 2678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:40:29.101370 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:40:29.101375 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:40:29.101377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:40:29.101522 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:40:29.101833 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:40:29.101837 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:40:29.101838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:40:29.102183 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:40:29.102439 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:29.102661 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 23:40:29.102842 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 23:40:29.107918 17567 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38748:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:40:29.115176 17567 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:40:29.140662 18038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:29.127895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38123 (local address 127.0.0.1:60540)
0620 23:40:29.129844 (+ 1949us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:29.129899 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 23:40:29.130147 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:40:29.131080 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:40:29.131092 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:40:29.132577 (+ 1485us) spinlock_profiling.cc:243] Waited 996 us on lock 0x61200002c928. stack: 00007fc46277266e 00007fc462772430 00007fc4617f95ed 00000000008ea064 00000000008ea02d 00000000008e99fa 00007fc4627680fd 00007fc462767d1e 00007fc4756713cd 00007fc47567136c 00007fc47563db3b 00007fc4691798e9 00007fc469175e45 00007fc4691fddca 00007fc4691fbf68 00007fc46922f82f
0620 23:40:29.132611 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:40:29.133107 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:29.133121 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:29.135062 (+ 1941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:29.135068 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:29.137544 (+ 2476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:29.137559 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:29.139622 (+ 2063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:29.139629 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:29.139859 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:29.139917 (+ 58us) client_negotiation.cc:769] Sending connection context
0620 23:40:29.140042 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 23:40:29.140260 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1619,"mutex_wait_us":1,"spinlock_wait_cycles":2289920,"thread_start_us":2589,"threads_started":1}
I20260620 23:40:29.141465 18037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:29.128162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60540 (local address 127.0.97.1:38123)
0620 23:40:29.128868 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:29.128879 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:29.130222 (+ 1343us) server_negotiation.cc:407] Connection header received
0620 23:40:29.130375 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:29.130387 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:29.130736 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:29.131376 (+ 640us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:40:29.133269 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:29.134793 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:29.137737 (+ 2944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:29.139139 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:29.139278 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:29.139541 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:29.140950 (+ 1409us) server_negotiation.cc:299] Negotiation successful
0620 23:40:29.141130 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":767,"threads_started":1}
I20260620 23:40:29.141503 18041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:29.131652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:32797 (local address 127.0.0.1:39256)
0620 23:40:29.132996 (+ 1344us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:29.133014 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:40:29.133154 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:40:29.134337 (+ 1183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:40:29.134341 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:40:29.134365 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:40:29.134699 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:29.134708 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:29.136448 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:29.136455 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:29.138451 (+ 1996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:29.138467 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:29.140582 (+ 2115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:29.140589 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:29.140817 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:29.140854 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 23:40:29.140945 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 23:40:29.141092 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1221,"mutex_wait_us":118,"thread_start_us":1067,"threads_started":1}
I20260620 23:40:29.141774 18039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:29.129216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32846 (local address 127.0.97.2:45495)
0620 23:40:29.132065 (+ 2849us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:29.132078 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:29.133302 (+ 1224us) server_negotiation.cc:407] Connection header received
0620 23:40:29.133396 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:29.133402 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:29.133600 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:29.133751 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:40:29.134390 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:29.136309 (+ 1919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:29.138559 (+ 2250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:29.140054 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:29.140178 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:29.140373 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:29.141277 (+ 904us) server_negotiation.cc:299] Negotiation successful
0620 23:40:29.141448 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2693,"thread_start_us":2588,"threads_started":1}
I20260620 23:40:29.142438 18040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:29.131271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39256 (local address 127.0.97.3:32797)
0620 23:40:29.132537 (+ 1266us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:29.132546 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:29.133125 (+ 579us) server_negotiation.cc:407] Connection header received
0620 23:40:29.134000 (+ 875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:29.134006 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:29.134157 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:29.134353 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:40:29.134904 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:29.136309 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:29.138628 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:29.140050 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:29.140178 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:29.140399 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:29.141940 (+ 1541us) server_negotiation.cc:299] Negotiation successful
0620 23:40:29.142160 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1074,"thread_start_us":902,"threads_started":1}
I20260620 23:40:29.141904 18036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:29.128672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45495 (local address 127.0.0.1:32846)
0620 23:40:29.130735 (+ 2063us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:29.130755 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:40:29.132142 (+ 1387us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:40:29.133781 (+ 1639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:40:29.133786 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:40:29.133812 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:40:29.134173 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:29.134184 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:29.136470 (+ 2286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:29.136475 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:29.138415 (+ 1940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:29.138426 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:29.140566 (+ 2140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:29.140574 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:29.141092 (+ 518us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:29.141122 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 23:40:29.141216 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 23:40:29.141341 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1945,"mutex_wait_us":338,"thread_start_us":673,"threads_started":1}
I20260620 23:40:29.145744 17691 tablet_service.cc:1467] Processing CreateTablet for tablet bce734aa35e0449ab64a15c02229f21b (DEFAULT_TABLE table=test-table [id=33b3e380c7e4494abe95d8b0b5d36440]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:40:29.145991 17959 tablet_service.cc:1467] Processing CreateTablet for tablet bce734aa35e0449ab64a15c02229f21b (DEFAULT_TABLE table=test-table [id=33b3e380c7e4494abe95d8b0b5d36440]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:40:29.146080 17825 tablet_service.cc:1467] Processing CreateTablet for tablet bce734aa35e0449ab64a15c02229f21b (DEFAULT_TABLE table=test-table [id=33b3e380c7e4494abe95d8b0b5d36440]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:40:29.147287 17959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bce734aa35e0449ab64a15c02229f21b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:40:29.147320 17825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bce734aa35e0449ab64a15c02229f21b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:40:29.147310 17691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bce734aa35e0449ab64a15c02229f21b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:40:29.160000 18042 tablet_bootstrap.cc:492] T bce734aa35e0449ab64a15c02229f21b P db19a538c9b54628858e8def3f84ed34: Bootstrap starting.
I20260620 23:40:29.163308 18042 tablet_bootstrap.cc:654] T bce734aa35e0449ab64a15c02229f21b P db19a538c9b54628858e8def3f84ed34: Neither blocks nor log segments found. Creating new log.
I20260620 23:40:29.163601 18043 tablet_bootstrap.cc:492] T bce734aa35e0449ab64a15c02229f21b P 3e7374a72999460ebbd606e772206d25: Bootstrap starting.
I20260620 23:40:29.164892 18042 log.cc:824] T bce734aa35e0449ab64a15c02229f21b P db19a538c9b54628858e8def3f84ed34: Log is configured to *not* fsync() on all Append() calls
I20260620 23:40:29.166383 18044 tablet_bootstrap.cc:492] T bce734aa35e0449ab64a15c02229f21b P aa54fe804793430289a9e7b0163f24ec: Bootstrap starting.
I20260620 23:40:29.167454 18043 tablet_bootstrap.cc:654] T bce734aa35e0449ab64a15c02229f21b P 3e7374a72999460ebbd606e772206d25: Neither blocks nor log segments found. Creating new log.
I20260620 23:40:29.168619 18042 tablet_bootstrap.cc:492] T bce734aa35e0449ab64a15c02229f21b P db19a538c9b54628858e8def3f84ed34: No bootstrap required, opened a new log
I20260620 23:40:29.168628 18043 log.cc:824] T bce734aa35e0449ab64a15c02229f21b P 3e7374a72999460ebbd606e772206d25: Log is configured to *not* fsync() on all Append() calls
I20260620 23:40:29.168910 18042 ts_tablet_manager.cc:1397] T bce734aa35e0449ab64a15c02229f21b P db19a538c9b54628858e8def3f84ed34: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260620 23:40:29.171262 18043 tablet_bootstrap.cc:492] T bce734aa35e0449ab64a15c02229f21b P 3e7374a72999460ebbd606e772206d25: No bootstrap required, opened a new log
I20260620 23:40:29.171481 18044 tablet_bootstrap.cc:654] T bce734aa35e0449ab64a15c02229f21b P aa54fe804793430289a9e7b0163f24ec: Neither blocks nor log segments found. Creating new log.
I20260620 23:40:29.171518 18043 ts_tablet_manager.cc:1397] T bce734aa35e0449ab64a15c02229f21b P 3e7374a72999460ebbd606e772206d25: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20260620 23:40:29.172683 18044 log.cc:824] T bce734aa35e0449ab64a15c02229f21b P aa54fe804793430289a9e7b0163f24ec: Log is configured to *not* fsync() on all Append() calls
I20260620 23:40:29.175777 18044 tablet_bootstrap.cc:492] T bce734aa35e0449ab64a15c02229f21b P aa54fe804793430289a9e7b0163f24ec: No bootstrap required, opened a new log
I20260620 23:40:29.176044 18044 ts_tablet_manager.cc:1397] T bce734aa35e0449ab64a15c02229f21b P aa54fe804793430289a9e7b0163f24ec: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260620 23:40:29.179416 18042 raft_consensus.cc:348] T bce734aa35e0449ab64a15c02229f21b P db19a538c9b54628858e8def3f84ed34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e7374a72999460ebbd606e772206d25" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38123 } } peers { permanent_uuid: "db19a538c9b54628858e8def3f84ed34" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32797 } } peers { permanent_uuid: "aa54fe804793430289a9e7b0163f24ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45495 } }
I20260620 23:40:29.179591 18043 raft_consensus.cc:348] T bce734aa35e0449ab64a15c02229f21b P 3e7374a72999460ebbd606e772206d25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e7374a72999460ebbd606e772206d25" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38123 } } peers { permanent_uuid: "db19a538c9b54628858e8def3f84ed34" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32797 } } peers { permanent_uuid: "aa54fe804793430289a9e7b0163f24ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45495 } }
I20260620 23:40:29.179883 18042 raft_consensus.cc:374] T bce734aa35e0449ab64a15c02229f21b P db19a538c9b54628858e8def3f84ed34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:40:29.179960 18043 raft_consensus.cc:374] T bce734aa35e0449ab64a15c02229f21b P 3e7374a72999460ebbd606e772206d25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:40:29.179966 18042 raft_consensus.cc:729] T bce734aa35e0449ab64a15c02229f21b P db19a538c9b54628858e8def3f84ed34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db19a538c9b54628858e8def3f84ed34, State: Initialized, Role: FOLLOWER
I20260620 23:40:29.180078 18043 raft_consensus.cc:729] T bce734aa35e0449ab64a15c02229f21b P 3e7374a72999460ebbd606e772206d25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e7374a72999460ebbd606e772206d25, State: Initialized, Role: FOLLOWER
I20260620 23:40:29.180621 18043 consensus_queue.cc:260] T bce734aa35e0449ab64a15c02229f21b P 3e7374a72999460ebbd606e772206d25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e7374a72999460ebbd606e772206d25" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38123 } } peers { permanent_uuid: "db19a538c9b54628858e8def3f84ed34" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32797 } } peers { permanent_uuid: "aa54fe804793430289a9e7b0163f24ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45495 } }
I20260620 23:40:29.180594 18042 consensus_queue.cc:260] T bce734aa35e0449ab64a15c02229f21b P db19a538c9b54628858e8def3f84ed34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e7374a72999460ebbd606e772206d25" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38123 } } peers { permanent_uuid: "db19a538c9b54628858e8def3f84ed34" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32797 } } peers { permanent_uuid: "aa54fe804793430289a9e7b0163f24ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45495 } }
I20260620 23:40:29.182269 18024 heartbeater.cc:503] Master 127.0.97.62:44793 was elected leader, sending a full tablet report...
I20260620 23:40:29.182526 18043 ts_tablet_manager.cc:1428] T bce734aa35e0449ab64a15c02229f21b P 3e7374a72999460ebbd606e772206d25: Time spent starting tablet: real 0.011s user 0.006s sys 0.004s
I20260620 23:40:29.182711 18042 ts_tablet_manager.cc:1428] T bce734aa35e0449ab64a15c02229f21b P db19a538c9b54628858e8def3f84ed34: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260620 23:40:29.186506 18044 raft_consensus.cc:348] T bce734aa35e0449ab64a15c02229f21b P aa54fe804793430289a9e7b0163f24ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e7374a72999460ebbd606e772206d25" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38123 } } peers { permanent_uuid: "db19a538c9b54628858e8def3f84ed34" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32797 } } peers { permanent_uuid: "aa54fe804793430289a9e7b0163f24ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45495 } }
I20260620 23:40:29.187021 18044 raft_consensus.cc:374] T bce734aa35e0449ab64a15c02229f21b P aa54fe804793430289a9e7b0163f24ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:40:29.187134 18044 raft_consensus.cc:729] T bce734aa35e0449ab64a15c02229f21b P aa54fe804793430289a9e7b0163f24ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa54fe804793430289a9e7b0163f24ec, State: Initialized, Role: FOLLOWER
I20260620 23:40:29.187808 18044 consensus_queue.cc:260] T bce734aa35e0449ab64a15c02229f21b P aa54fe804793430289a9e7b0163f24ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e7374a72999460ebbd606e772206d25" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38123 } } peers { permanent_uuid: "db19a538c9b54628858e8def3f84ed34" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32797 } } peers { permanent_uuid: "aa54fe804793430289a9e7b0163f24ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45495 } }
I20260620 23:40:29.189495 18044 ts_tablet_manager.cc:1428] T bce734aa35e0449ab64a15c02229f21b P aa54fe804793430289a9e7b0163f24ec: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260620 23:40:29.189747 17890 heartbeater.cc:503] Master 127.0.97.62:44793 was elected leader, sending a full tablet report...
I20260620 23:40:29.190188 18048 raft_consensus.cc:482] T bce734aa35e0449ab64a15c02229f21b P 3e7374a72999460ebbd606e772206d25 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:40:29.190482 18048 raft_consensus.cc:504] T bce734aa35e0449ab64a15c02229f21b P 3e7374a72999460ebbd606e772206d25 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e7374a72999460ebbd606e772206d25" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38123 } } peers { permanent_uuid: "db19a538c9b54628858e8def3f84ed34" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32797 } } peers { permanent_uuid: "aa54fe804793430289a9e7b0163f24ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45495 } }
I20260620 23:40:29.196192 18048 leader_election.cc:288] T bce734aa35e0449ab64a15c02229f21b P 3e7374a72999460ebbd606e772206d25 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db19a538c9b54628858e8def3f84ed34 (127.0.97.3:32797), aa54fe804793430289a9e7b0163f24ec (127.0.97.2:45495)
I20260620 23:40:29.200624 18040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:29.194422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41373 (local address 127.0.97.3:32797)
0620 23:40:29.194612 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:29.194621 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:29.195042 (+ 421us) server_negotiation.cc:407] Connection header received
0620 23:40:29.195257 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:29.195262 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:29.195336 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:29.195434 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:40:29.196420 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:29.197140 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:29.198879 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:29.199708 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:29.199853 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:29.199965 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:29.200237 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 23:40:29.200363 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 23:40:29.200763 18052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:29.194025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:32797 (local address 127.0.97.1:41373)
0620 23:40:29.194925 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:29.194970 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 23:40:29.195109 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:40:29.195484 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:40:29.195489 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:40:29.195521 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:40:29.196251 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:29.196264 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:29.197274 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:29.197282 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:29.198725 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:29.198737 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:29.199919 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:29.199925 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:29.200083 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:29.200106 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 23:40:29.200211 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 23:40:29.200348 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":773,"mutex_wait_us":3,"thread_start_us":626,"threads_started":1}
I20260620 23:40:29.201162 18039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:29.195492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45033 (local address 127.0.97.2:45495)
0620 23:40:29.195686 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:29.195693 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:29.196286 (+ 593us) server_negotiation.cc:407] Connection header received
0620 23:40:29.196454 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:29.196458 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:29.196523 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:29.196610 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:40:29.197068 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:29.197732 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:29.199494 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:29.200253 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:29.200410 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:29.200508 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:29.200833 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 23:40:29.200975 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 23:40:29.201870 18053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:29.195473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45495 (local address 127.0.97.1:45033)
0620 23:40:29.196187 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:29.196202 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:40:29.196344 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:40:29.196624 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:40:29.196628 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:40:29.196650 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:40:29.196938 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:29.196945 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:29.197869 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:29.197875 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:29.199348 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:29.199359 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:29.200485 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:29.200492 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:29.200626 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:29.200648 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 23:40:29.201354 (+ 706us) client_negotiation.cc:240] Negotiation successful
0620 23:40:29.201491 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"mutex_wait_us":20,"thread_start_us":464,"threads_started":1}
I20260620 23:40:29.201761 17979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bce734aa35e0449ab64a15c02229f21b" candidate_uuid: "3e7374a72999460ebbd606e772206d25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db19a538c9b54628858e8def3f84ed34" is_pre_election: true
I20260620 23:40:29.202401 17979 raft_consensus.cc:2454] T bce734aa35e0449ab64a15c02229f21b P db19a538c9b54628858e8def3f84ed34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e7374a72999460ebbd606e772206d25 in term 0.
I20260620 23:40:29.203099 17644 leader_election.cc:302] T bce734aa35e0449ab64a15c02229f21b P 3e7374a72999460ebbd606e772206d25 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e7374a72999460ebbd606e772206d25, db19a538c9b54628858e8def3f84ed34; no voters:
I20260620 23:40:29.202967 17845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bce734aa35e0449ab64a15c02229f21b" candidate_uuid: "3e7374a72999460ebbd606e772206d25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa54fe804793430289a9e7b0163f24ec" is_pre_election: true
I20260620 23:40:29.203469 18048 raft_consensus.cc:2781] T bce734aa35e0449ab64a15c02229f21b P 3e7374a72999460ebbd606e772206d25 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:40:29.203473 17845 raft_consensus.cc:2454] T bce734aa35e0449ab64a15c02229f21b P aa54fe804793430289a9e7b0163f24ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e7374a72999460ebbd606e772206d25 in term 0.
I20260620 23:40:29.203596 18048 raft_consensus.cc:482] T bce734aa35e0449ab64a15c02229f21b P 3e7374a72999460ebbd606e772206d25 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:40:29.203708 18048 raft_consensus.cc:3037] T bce734aa35e0449ab64a15c02229f21b P 3e7374a72999460ebbd606e772206d25 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:40:29.205976 18048 raft_consensus.cc:504] T bce734aa35e0449ab64a15c02229f21b P 3e7374a72999460ebbd606e772206d25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e7374a72999460ebbd606e772206d25" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38123 } } peers { permanent_uuid: "db19a538c9b54628858e8def3f84ed34" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32797 } } peers { permanent_uuid: "aa54fe804793430289a9e7b0163f24ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45495 } }
I20260620 23:40:29.206583 18048 leader_election.cc:288] T bce734aa35e0449ab64a15c02229f21b P 3e7374a72999460ebbd606e772206d25 [CANDIDATE]: Term 1 election: Requested vote from peers db19a538c9b54628858e8def3f84ed34 (127.0.97.3:32797), aa54fe804793430289a9e7b0163f24ec (127.0.97.2:45495)
I20260620 23:40:29.207073 17979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bce734aa35e0449ab64a15c02229f21b" candidate_uuid: "3e7374a72999460ebbd606e772206d25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db19a538c9b54628858e8def3f84ed34"
I20260620 23:40:29.207108 17845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bce734aa35e0449ab64a15c02229f21b" candidate_uuid: "3e7374a72999460ebbd606e772206d25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa54fe804793430289a9e7b0163f24ec"
I20260620 23:40:29.207307 17845 raft_consensus.cc:3037] T bce734aa35e0449ab64a15c02229f21b P aa54fe804793430289a9e7b0163f24ec [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:40:29.207288 17979 raft_consensus.cc:3037] T bce734aa35e0449ab64a15c02229f21b P db19a538c9b54628858e8def3f84ed34 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:40:29.209779 17845 raft_consensus.cc:2454] T bce734aa35e0449ab64a15c02229f21b P aa54fe804793430289a9e7b0163f24ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e7374a72999460ebbd606e772206d25 in term 1.
I20260620 23:40:29.209789 17979 raft_consensus.cc:2454] T bce734aa35e0449ab64a15c02229f21b P db19a538c9b54628858e8def3f84ed34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e7374a72999460ebbd606e772206d25 in term 1.
I20260620 23:40:29.210191 17644 leader_election.cc:302] T bce734aa35e0449ab64a15c02229f21b P 3e7374a72999460ebbd606e772206d25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e7374a72999460ebbd606e772206d25, db19a538c9b54628858e8def3f84ed34; no voters:
I20260620 23:40:29.210515 18048 raft_consensus.cc:2781] T bce734aa35e0449ab64a15c02229f21b P 3e7374a72999460ebbd606e772206d25 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:40:29.210690 18048 raft_consensus.cc:686] T bce734aa35e0449ab64a15c02229f21b P 3e7374a72999460ebbd606e772206d25 [term 1 LEADER]: Becoming Leader. State: Replica: 3e7374a72999460ebbd606e772206d25, State: Running, Role: LEADER
I20260620 23:40:29.211244 18048 consensus_queue.cc:237] T bce734aa35e0449ab64a15c02229f21b P 3e7374a72999460ebbd606e772206d25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e7374a72999460ebbd606e772206d25" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38123 } } peers { permanent_uuid: "db19a538c9b54628858e8def3f84ed34" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32797 } } peers { permanent_uuid: "aa54fe804793430289a9e7b0163f24ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45495 } }
I20260620 23:40:29.215915 17567 catalog_manager.cc:5495] T bce734aa35e0449ab64a15c02229f21b P 3e7374a72999460ebbd606e772206d25 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e7374a72999460ebbd606e772206d25 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "3e7374a72999460ebbd606e772206d25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e7374a72999460ebbd606e772206d25" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db19a538c9b54628858e8def3f84ed34" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa54fe804793430289a9e7b0163f24ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45495 } health_report { overall_health: UNKNOWN } } }
I20260620 23:40:29.240509 18037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:29.234826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60556 (local address 127.0.97.1:38123)
0620 23:40:29.235061 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:29.235068 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:29.235167 (+ 99us) server_negotiation.cc:407] Connection header received
0620 23:40:29.235432 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:29.235436 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:29.235531 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:29.235687 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:40:29.236097 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:29.236829 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:29.238233 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:29.238640 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:29.238775 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:29.239347 (+ 572us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:40:29.239862 (+ 515us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:40:29.240076 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:29.240147 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 23:40:29.240234 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 23:40:29.249253 17979 raft_consensus.cc:1261] T bce734aa35e0449ab64a15c02229f21b P db19a538c9b54628858e8def3f84ed34 [term 1 FOLLOWER]: Refusing update from remote peer 3e7374a72999460ebbd606e772206d25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:40:29.249253 17845 raft_consensus.cc:1261] T bce734aa35e0449ab64a15c02229f21b P aa54fe804793430289a9e7b0163f24ec [term 1 FOLLOWER]: Refusing update from remote peer 3e7374a72999460ebbd606e772206d25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:40:29.250386 18050 consensus_queue.cc:1035] T bce734aa35e0449ab64a15c02229f21b P 3e7374a72999460ebbd606e772206d25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db19a538c9b54628858e8def3f84ed34" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:40:29.250800 18048 consensus_queue.cc:1035] T bce734aa35e0449ab64a15c02229f21b P 3e7374a72999460ebbd606e772206d25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa54fe804793430289a9e7b0163f24ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 23:40:29.253057 17757 tablet.cc:2369] T bce734aa35e0449ab64a15c02229f21b P 3e7374a72999460ebbd606e772206d25: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:40:29.262869 18056 mvcc.cc:205] Tried to move back new op lower bound from 7299067204593500160 to 7299067204453298176. Current Snapshot: MvccSnapshot[applied={T|T < 7299067204593500160}]
I20260620 23:40:29.263788 18058 mvcc.cc:205] Tried to move back new op lower bound from 7299067204593500160 to 7299067204453298176. Current Snapshot: MvccSnapshot[applied={T|T < 7299067204593500160}]
I20260620 23:40:29.277546 17567 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:40:29.277866 17567 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:40:29.281906 17567 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3303509587744568af792ad41b20da1e: Sending DeleteTablet for 3 replicas of tablet bce734aa35e0449ab64a15c02229f21b
I20260620 23:40:29.283315 17825 tablet_service.cc:1514] Processing DeleteTablet for tablet bce734aa35e0449ab64a15c02229f21b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:40:29 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:32846
I20260620 23:40:29.283654 17691 tablet_service.cc:1514] Processing DeleteTablet for tablet bce734aa35e0449ab64a15c02229f21b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:40:29 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60540
I20260620 23:40:29.283700 17959 tablet_service.cc:1514] Processing DeleteTablet for tablet bce734aa35e0449ab64a15c02229f21b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:40:29 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39256
I20260620 23:40:29.285576 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 17628
I20260620 23:40:29.296926 18067 tablet_replica.cc:330] T bce734aa35e0449ab64a15c02229f21b P 3e7374a72999460ebbd606e772206d25: stopping tablet replica
I20260620 23:40:29.297636 18068 tablet_replica.cc:330] T bce734aa35e0449ab64a15c02229f21b P db19a538c9b54628858e8def3f84ed34: stopping tablet replica
I20260620 23:40:29.298012 18066 tablet_replica.cc:330] T bce734aa35e0449ab64a15c02229f21b P aa54fe804793430289a9e7b0163f24ec: stopping tablet replica
I20260620 23:40:29.298497 18066 raft_consensus.cc:2229] T bce734aa35e0449ab64a15c02229f21b P aa54fe804793430289a9e7b0163f24ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:40:29.299036 18067 raft_consensus.cc:2229] T bce734aa35e0449ab64a15c02229f21b P 3e7374a72999460ebbd606e772206d25 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:40:29.298561 18068 raft_consensus.cc:2229] T bce734aa35e0449ab64a15c02229f21b P db19a538c9b54628858e8def3f84ed34 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:40:29.299196 18066 raft_consensus.cc:2258] T bce734aa35e0449ab64a15c02229f21b P aa54fe804793430289a9e7b0163f24ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:40:29.300916 18067 raft_consensus.cc:2258] T bce734aa35e0449ab64a15c02229f21b P 3e7374a72999460ebbd606e772206d25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:40:29.301204 18068 raft_consensus.cc:2258] T bce734aa35e0449ab64a15c02229f21b P db19a538c9b54628858e8def3f84ed34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:40:29.301679 18066 ts_tablet_manager.cc:1905] T bce734aa35e0449ab64a15c02229f21b P aa54fe804793430289a9e7b0163f24ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:40:29.304680 18068 ts_tablet_manager.cc:1905] T bce734aa35e0449ab64a15c02229f21b P db19a538c9b54628858e8def3f84ed34: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:40:29.306413 18067 ts_tablet_manager.cc:1905] T bce734aa35e0449ab64a15c02229f21b P 3e7374a72999460ebbd606e772206d25: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 23:40:29 dist-test-slave-tcgx krb5kdc[17522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 23:40:29.310637 18066 ts_tablet_manager.cc:1918] T bce734aa35e0449ab64a15c02229f21b P aa54fe804793430289a9e7b0163f24ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:40:29.310837 18066 log.cc:1196] T bce734aa35e0449ab64a15c02229f21b P aa54fe804793430289a9e7b0163f24ec: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/minicluster-data/ts-1/wal/wals/bce734aa35e0449ab64a15c02229f21b
I20260620 23:40:29.311502 18066 ts_tablet_manager.cc:1939] T bce734aa35e0449ab64a15c02229f21b P aa54fe804793430289a9e7b0163f24ec: Deleting consensus metadata
I20260620 23:40:29.313747 17552 catalog_manager.cc:4841] TS aa54fe804793430289a9e7b0163f24ec (127.0.97.2:45495): tablet bce734aa35e0449ab64a15c02229f21b (table test-table [id=33b3e380c7e4494abe95d8b0b5d36440]) successfully deleted
I20260620 23:40:29.319131 18037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:29.287323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60560 (local address 127.0.97.1:38123)
0620 23:40:29.287508 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:29.287515 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:29.291592 (+ 4077us) server_negotiation.cc:407] Connection header received
0620 23:40:29.291970 (+ 378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:29.291980 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:29.292102 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:29.292222 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:40:29.297837 (+ 5615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:29.299261 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:29.301553 (+ 2292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:29.302342 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:29.302454 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:29.309169 (+ 6715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:40:29.309188 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:40:29.309207 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:40:29.309282 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:40:29.314145 (+ 4863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:40:29.315446 (+ 1301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:40:29.315458 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:40:29.315463 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:40:29.315807 (+ 344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:40:29.316417 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:40:29.316427 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:40:29.316431 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:40:29.317082 (+ 651us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:40:29.317512 (+ 430us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:29.318264 (+ 752us) server_negotiation.cc:299] Negotiation successful
0620 23:40:29.318636 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 23:40:29.320132 18068 ts_tablet_manager.cc:1918] T bce734aa35e0449ab64a15c02229f21b P db19a538c9b54628858e8def3f84ed34: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:40:29.320398 18068 log.cc:1196] T bce734aa35e0449ab64a15c02229f21b P db19a538c9b54628858e8def3f84ed34: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/minicluster-data/ts-2/wal/wals/bce734aa35e0449ab64a15c02229f21b
I20260620 23:40:29.320888 17751 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:40:29.321009 18068 ts_tablet_manager.cc:1939] T bce734aa35e0449ab64a15c02229f21b P db19a538c9b54628858e8def3f84ed34: Deleting consensus metadata
I20260620 23:40:29.323467 17551 catalog_manager.cc:4841] TS db19a538c9b54628858e8def3f84ed34 (127.0.97.3:32797): tablet bce734aa35e0449ab64a15c02229f21b (table test-table [id=33b3e380c7e4494abe95d8b0b5d36440]) successfully deleted
I20260620 23:40:29.424454 18067 ts_tablet_manager.cc:1918] T bce734aa35e0449ab64a15c02229f21b P 3e7374a72999460ebbd606e772206d25: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:40:29.424736 18067 log.cc:1196] T bce734aa35e0449ab64a15c02229f21b P 3e7374a72999460ebbd606e772206d25: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998525501163-388-0/minicluster-data/ts-0/wal/wals/bce734aa35e0449ab64a15c02229f21b
I20260620 23:40:29.425033 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 17628
W20260620 23:40:29.443879 17551 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:38123 (error 108)
I20260620 23:40:29.444128 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 17762
I20260620 23:40:29.445791 18036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:29.445315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38123 (local address 127.0.0.1:60568)
0620 23:40:29.445467 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:29.445605 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 23:40:29.446045 17551 catalog_manager.cc:4584] TS 3e7374a72999460ebbd606e772206d25 (127.0.97.1:38123): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bce734aa35e0449ab64a15c02229f21b: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38123: connect: Connection refused (error 111)
Jun 20 23:40:29 dist-test-slave-tcgx krb5kdc[17522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 23:40:29.460214 18039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:29.444869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32852 (local address 127.0.97.2:45495)
0620 23:40:29.445048 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:29.445052 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:29.445124 (+ 72us) server_negotiation.cc:407] Connection header received
0620 23:40:29.445333 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:29.445340 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:29.445425 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:29.445544 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:40:29.446836 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:29.447642 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:29.448916 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:29.449308 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:29.449429 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:29.453574 (+ 4145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:40:29.453590 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:40:29.453644 (+ 54us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:40:29.453683 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:40:29.457191 (+ 3508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:40:29.458006 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:40:29.458014 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:40:29.458016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:40:29.458150 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:40:29.458472 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:40:29.458476 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:40:29.458478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:40:29.458982 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:40:29.459273 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:29.459619 (+ 346us) server_negotiation.cc:299] Negotiation successful
0620 23:40:29.459843 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 23:40:29.460933 17885 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:40:29.499794 18036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:29.499294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38123 (local address 127.0.0.1:60582)
0620 23:40:29.499509 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:29.499617 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260620 23:40:29.500890 18036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:29.500549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38123 (local address 127.0.0.1:60594)
0620 23:40:29.500687 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:29.500762 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 23:40:29.554188 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 17762
I20260620 23:40:29.561375 18036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:29.560920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38123 (local address 127.0.0.1:60608)
0620 23:40:29.561136 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:29.561217 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 23:40:29.562340 18036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:29.562010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38123 (local address 127.0.0.1:60618)
0620 23:40:29.562148 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:29.562217 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 23:40:29.571225 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 17896
Jun 20 23:40:29 dist-test-slave-tcgx krb5kdc[17522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 23:40:29.586004 18040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:29.572021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39266 (local address 127.0.97.3:32797)
0620 23:40:29.572226 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:29.572232 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:29.572254 (+ 22us) server_negotiation.cc:407] Connection header received
0620 23:40:29.572327 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:29.572333 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:29.572404 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:29.572510 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:40:29.573837 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:29.574583 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:29.575920 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:29.576326 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:29.576452 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:29.580839 (+ 4387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:40:29.580854 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:40:29.580872 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:40:29.580914 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:40:29.583667 (+ 2753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:40:29.584290 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:40:29.584296 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:40:29.584299 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:40:29.584426 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:40:29.584747 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:40:29.584750 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:40:29.584752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:40:29.585100 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:40:29.585311 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:29.585572 (+ 261us) server_negotiation.cc:299] Negotiation successful
0620 23:40:29.585730 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 23:40:29.586644 18019 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:40:29.643003 18036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:29.642446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38123 (local address 127.0.0.1:60634)
0620 23:40:29.642684 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:29.642801 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260620 23:40:29.644120 18036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:29.643694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38123 (local address 127.0.0.1:60650)
0620 23:40:29.643938 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:29.644007 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260620 23:40:29.678426 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 17896
I20260620 23:40:29.695209 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 17535
I20260620 23:40:29.696436 17597 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:40:29.808293 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 17535
2026-06-20T23:40:29Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4273 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T23:40:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:40:29Z Disabled control of system clock
I20260620 23:40:29.886845 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998525501163-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998525501163-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998525501163-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43365
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:38661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43365
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 23:40:30.225538 18078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:40:30.225865 18078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:40:30.225955 18078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:40:30.233359 18078 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:40:30.233484 18078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:40:30.233525 18078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:40:30.233574 18078 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:40:30.233620 18078 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:40:30.243112 18078 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998525501163-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998525501163-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43365
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998525501163-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:40:30.244560 18078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:40:30.246713 18078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:40:30.258683 18086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:40:30.258764 18084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:40:30.258724 18083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:40:30.259757 18078 server_base.cc:1034] running on GCE node
I20260620 23:40:30.260520 18078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:40:30.261936 18078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:40:30.263130 18078 hybrid_clock.cc:648] HybridClock initialized: now 1781998830263062 us; error 56 us; skew 500 ppm
I20260620 23:40:30.263504 18078 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:40:30.269542 18078 webserver.cc:460] Webserver started at http://127.0.97.62:37451/ using document root <none> and password file <none>
I20260620 23:40:30.270399 18078 fs_manager.cc:359] Metadata directory not provided
I20260620 23:40:30.270519 18078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:40:30.270871 18078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:40:30.273917 18078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998525501163-388-0/minicluster-data/master-0/data/instance:
uuid: "fc66e800bb0f44f9af07d6ee58230318"
format_stamp: "Formatted at 2026-06-20 23:40:30 on dist-test-slave-tcgx"
server_key: "c61720206c33a390af3a5261e78adc24"
server_key_iv: "8f38929758463827a917b210829f943e"
server_key_version: "encryptionkey@0"
I20260620 23:40:30.274751 18078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998525501163-388-0/minicluster-data/master-0/wal/instance:
uuid: "fc66e800bb0f44f9af07d6ee58230318"
format_stamp: "Formatted at 2026-06-20 23:40:30 on dist-test-slave-tcgx"
server_key: "c61720206c33a390af3a5261e78adc24"
server_key_iv: "8f38929758463827a917b210829f943e"
server_key_version: "encryptionkey@0"
I20260620 23:40:30.280126 18078 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 23:40:30.283820 18092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:30.285327 18078 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:40:30.285535 18078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998525501163-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998525501163-388-0/minicluster-data/master-0/wal
uuid: "fc66e800bb0f44f9af07d6ee58230318"
format_stamp: "Formatted at 2026-06-20 23:40:30 on dist-test-slave-tcgx"
server_key: "c61720206c33a390af3a5261e78adc24"
server_key_iv: "8f38929758463827a917b210829f943e"
server_key_version: "encryptionkey@0"
I20260620 23:40:30.285774 18078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998525501163-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998525501163-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998525501163-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:40:30.318809 18078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:40:30.319808 18078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:40:30.320111 18078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:40:30.338860 18078 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43365
I20260620 23:40:30.338877 18143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43365 every 8 connection(s)
I20260620 23:40:30.340298 18078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:40:30.341545 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 18078
I20260620 23:40:30.341740 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998525501163-388-0/minicluster-data/master-0/wal/instance
I20260620 23:40:30.342298 388 external_mini_cluster.cc:1428] Setting key ec3d0a0a461989ba8510784bcda0f60e
I20260620 23:40:30.345263 18144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:40:30.356227 18147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:30.344413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58812 (local address 127.0.97.62:43365)
0620 23:40:30.345469 (+ 1056us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:30.345481 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:30.345522 (+ 41us) server_negotiation.cc:407] Connection header received
0620 23:40:30.347231 (+ 1709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:30.347262 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:30.347756 (+ 494us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:30.348163 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:40:30.348735 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:30.350126 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:30.351625 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:30.352310 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:30.352504 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:30.353702 (+ 1198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:40:30.353709 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:40:30.353725 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:40:30.354394 (+ 669us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:40:30.354486 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:40:30.354508 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:40:30.354638 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:30.354865 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 23:40:30.355267 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":371,"threads_started":1}
I20260620 23:40:30.356575 18144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc66e800bb0f44f9af07d6ee58230318: Bootstrap starting.
I20260620 23:40:30.359840 18144 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc66e800bb0f44f9af07d6ee58230318: Neither blocks nor log segments found. Creating new log.
I20260620 23:40:30.360836 18144 log.cc:824] T 00000000000000000000000000000000 P fc66e800bb0f44f9af07d6ee58230318: Log is configured to *not* fsync() on all Append() calls
I20260620 23:40:30.363682 18144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc66e800bb0f44f9af07d6ee58230318: No bootstrap required, opened a new log
I20260620 23:40:30.371897 18144 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc66e800bb0f44f9af07d6ee58230318 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc66e800bb0f44f9af07d6ee58230318" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43365 } }
I20260620 23:40:30.372275 18144 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc66e800bb0f44f9af07d6ee58230318 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:40:30.372401 18144 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc66e800bb0f44f9af07d6ee58230318 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc66e800bb0f44f9af07d6ee58230318, State: Initialized, Role: FOLLOWER
I20260620 23:40:30.373035 18144 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc66e800bb0f44f9af07d6ee58230318 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc66e800bb0f44f9af07d6ee58230318" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43365 } }
I20260620 23:40:30.373306 18144 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc66e800bb0f44f9af07d6ee58230318 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:40:30.373438 18144 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc66e800bb0f44f9af07d6ee58230318 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:40:30.373623 18144 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc66e800bb0f44f9af07d6ee58230318 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:40:30.376292 18144 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc66e800bb0f44f9af07d6ee58230318 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc66e800bb0f44f9af07d6ee58230318" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43365 } }
I20260620 23:40:30.376833 18144 leader_election.cc:302] T 00000000000000000000000000000000 P fc66e800bb0f44f9af07d6ee58230318 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc66e800bb0f44f9af07d6ee58230318; no voters:
I20260620 23:40:30.377493 18144 leader_election.cc:288] T 00000000000000000000000000000000 P fc66e800bb0f44f9af07d6ee58230318 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:40:30.377693 18149 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc66e800bb0f44f9af07d6ee58230318 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:40:30.378463 18149 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc66e800bb0f44f9af07d6ee58230318 [term 1 LEADER]: Becoming Leader. State: Replica: fc66e800bb0f44f9af07d6ee58230318, State: Running, Role: LEADER
I20260620 23:40:30.379005 18149 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc66e800bb0f44f9af07d6ee58230318 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc66e800bb0f44f9af07d6ee58230318" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43365 } }
I20260620 23:40:30.379906 18144 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc66e800bb0f44f9af07d6ee58230318 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:40:30.382082 18151 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc66e800bb0f44f9af07d6ee58230318 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc66e800bb0f44f9af07d6ee58230318. Latest consensus state: current_term: 1 leader_uuid: "fc66e800bb0f44f9af07d6ee58230318" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc66e800bb0f44f9af07d6ee58230318" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43365 } } }
I20260620 23:40:30.382690 18151 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc66e800bb0f44f9af07d6ee58230318 [sys.catalog]: This master's current role is: LEADER
I20260620 23:40:30.383313 18150 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc66e800bb0f44f9af07d6ee58230318 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc66e800bb0f44f9af07d6ee58230318" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc66e800bb0f44f9af07d6ee58230318" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43365 } } }
I20260620 23:40:30.383737 18150 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc66e800bb0f44f9af07d6ee58230318 [sys.catalog]: This master's current role is: LEADER
I20260620 23:40:30.385200 18158 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:40:30.389765 18158 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:40:30.399346 18158 catalog_manager.cc:1347] Generated new cluster ID: 7cffd8e77fec437196400174544b54bf
I20260620 23:40:30.399475 18158 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:40:30.412343 18158 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:40:30.414222 18158 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:40:30.425992 18158 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc66e800bb0f44f9af07d6ee58230318: Generated new TSK 0
I20260620 23:40:30.427196 18158 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:40:30.436088 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998525501163-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998525501163-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998525501163-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43365
--builtin_ntp_servers=127.0.97.20:38661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 23:40:30.773880 18168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:40:30.774173 18168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:40:30.774267 18168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:40:30.781824 18168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:40:30.781941 18168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:40:30.782052 18168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 23:40:30.792006 18168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998525501163-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998525501163-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998525501163-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:40:30.793315 18168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:40:30.795120 18168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:40:30.808480 18173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:40:30.808445 18174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:40:30.808573 18176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:40:30.809561 18168 server_base.cc:1034] running on GCE node
I20260620 23:40:30.810173 18168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:40:30.811220 18168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:40:30.812448 18168 hybrid_clock.cc:648] HybridClock initialized: now 1781998830812397 us; error 44 us; skew 500 ppm
I20260620 23:40:30.812762 18168 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:40:30.815160 18168 webserver.cc:460] Webserver started at http://127.0.97.1:46095/ using document root <none> and password file <none>
I20260620 23:40:30.815867 18168 fs_manager.cc:359] Metadata directory not provided
I20260620 23:40:30.816000 18168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:40:30.816320 18168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:40:30.819263 18168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998525501163-388-0/minicluster-data/ts-0/data/instance:
uuid: "5663c9cb2afd4218a77060ada53e9ff1"
format_stamp: "Formatted at 2026-06-20 23:40:30 on dist-test-slave-tcgx"
server_key: "3e420e7be875748cf0a2989ac140a73a"
server_key_iv: "82ff6cb62755517a0163dea15db60c24"
server_key_version: "encryptionkey@0"
I20260620 23:40:30.820072 18168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998525501163-388-0/minicluster-data/ts-0/wal/instance:
uuid: "5663c9cb2afd4218a77060ada53e9ff1"
format_stamp: "Formatted at 2026-06-20 23:40:30 on dist-test-slave-tcgx"
server_key: "3e420e7be875748cf0a2989ac140a73a"
server_key_iv: "82ff6cb62755517a0163dea15db60c24"
server_key_version: "encryptionkey@0"
I20260620 23:40:30.825238 18168 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260620 23:40:30.828884 18182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:30.830453 18168 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:40:30.830694 18168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998525501163-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998525501163-388-0/minicluster-data/ts-0/wal
uuid: "5663c9cb2afd4218a77060ada53e9ff1"
format_stamp: "Formatted at 2026-06-20 23:40:30 on dist-test-slave-tcgx"
server_key: "3e420e7be875748cf0a2989ac140a73a"
server_key_iv: "82ff6cb62755517a0163dea15db60c24"
server_key_version: "encryptionkey@0"
I20260620 23:40:30.830969 18168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998525501163-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998525501163-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998525501163-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:40:30.876391 18168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:40:30.877359 18168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:40:30.877676 18168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:40:30.878787 18168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:40:30.880699 18168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:40:30.880827 18168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:30.880980 18168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:40:30.881085 18168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:30.916364 18168 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40587
I20260620 23:40:30.916417 18294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40587 every 8 connection(s)
I20260620 23:40:30.917809 18168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
I20260620 23:40:30.921012 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 18168
I20260620 23:40:30.921178 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998525501163-388-0/minicluster-data/ts-0/wal/instance
I20260620 23:40:30.921839 388 external_mini_cluster.cc:1428] Setting key 14682451c25f5ea6da88b2b0eb6a8d10
I20260620 23:40:30.924892 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998525501163-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998525501163-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998525501163-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43365
--builtin_ntp_servers=127.0.97.20:38661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 23:40:30.931919 18297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:30.920444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52051 (local address 127.0.97.62:43365)
0620 23:40:30.920986 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:30.920993 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:30.922484 (+ 1491us) server_negotiation.cc:407] Connection header received
0620 23:40:30.924855 (+ 2371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:30.924859 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:30.924969 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:30.925103 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:40:30.926576 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:30.927345 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:30.929048 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:30.929465 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:30.929595 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:30.931033 (+ 1438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:40:30.931037 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:40:30.931041 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:40:30.931061 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:40:30.931103 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:40:30.931115 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:40:30.931210 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:30.931442 (+ 232us) server_negotiation.cc:299] Negotiation successful
0620 23:40:30.931546 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":271,"threads_started":1}
I20260620 23:40:30.932981 18298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:30.921029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43365 (local address 127.0.97.1:52051)
0620 23:40:30.922283 (+ 1254us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:30.922332 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 23:40:30.924544 (+ 2212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:40:30.925517 (+ 973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:40:30.925529 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:40:30.925676 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:40:30.926409 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:30.926430 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:30.927496 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:30.927500 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:30.928884 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:30.928904 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:30.929706 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:30.929715 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:30.929949 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:30.930672 (+ 723us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:40:30.930674 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:40:30.930717 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:40:30.930806 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:40:30.931210 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:40:30.931224 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:40:30.931235 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 23:40:30.931433 (+ 198us) client_negotiation.cc:240] Negotiation successful
0620 23:40:30.931875 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":359,"threads_started":1}
I20260620 23:40:30.934612 18295 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43365
I20260620 23:40:30.935014 18295 heartbeater.cc:464] Registering TS with master...
I20260620 23:40:30.935994 18295 heartbeater.cc:511] Master 127.0.97.62:43365 requested a full tablet report, sending...
I20260620 23:40:30.938401 18109 ts_manager.cc:194] Registered new tserver with Master: 5663c9cb2afd4218a77060ada53e9ff1 (127.0.97.1:40587)
I20260620 23:40:30.940662 18109 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:52051
W20260620 23:40:31.260039 18299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:40:31.260327 18299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:40:31.260411 18299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:40:31.267912 18299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:40:31.268028 18299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:40:31.268134 18299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 23:40:31.277266 18299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998525501163-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998525501163-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998525501163-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:40:31.278505 18299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:40:31.280437 18299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:40:31.293890 18304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:40:31.293962 18307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:40:31.293939 18305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:40:31.294796 18299 server_base.cc:1034] running on GCE node
I20260620 23:40:31.295482 18299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:40:31.296440 18299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:40:31.297672 18299 hybrid_clock.cc:648] HybridClock initialized: now 1781998831297617 us; error 49 us; skew 500 ppm
I20260620 23:40:31.297995 18299 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:40:31.300428 18299 webserver.cc:460] Webserver started at http://127.0.97.2:41619/ using document root <none> and password file <none>
I20260620 23:40:31.301151 18299 fs_manager.cc:359] Metadata directory not provided
I20260620 23:40:31.301275 18299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:40:31.301597 18299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:40:31.304543 18299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998525501163-388-0/minicluster-data/ts-1/data/instance:
uuid: "be322151ed5f4a15875779fbd9a15693"
format_stamp: "Formatted at 2026-06-20 23:40:31 on dist-test-slave-tcgx"
server_key: "5ab3a1886330caf60147f805f461c247"
server_key_iv: "205f83325864bc78e155e443e99d68fc"
server_key_version: "encryptionkey@0"
I20260620 23:40:31.305348 18299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998525501163-388-0/minicluster-data/ts-1/wal/instance:
uuid: "be322151ed5f4a15875779fbd9a15693"
format_stamp: "Formatted at 2026-06-20 23:40:31 on dist-test-slave-tcgx"
server_key: "5ab3a1886330caf60147f805f461c247"
server_key_iv: "205f83325864bc78e155e443e99d68fc"
server_key_version: "encryptionkey@0"
I20260620 23:40:31.310478 18299 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 23:40:31.314098 18313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:31.315555 18299 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 23:40:31.315732 18299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998525501163-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998525501163-388-0/minicluster-data/ts-1/wal
uuid: "be322151ed5f4a15875779fbd9a15693"
format_stamp: "Formatted at 2026-06-20 23:40:31 on dist-test-slave-tcgx"
server_key: "5ab3a1886330caf60147f805f461c247"
server_key_iv: "205f83325864bc78e155e443e99d68fc"
server_key_version: "encryptionkey@0"
I20260620 23:40:31.315965 18299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998525501163-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998525501163-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998525501163-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:40:31.373977 18299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:40:31.374985 18299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:40:31.375299 18299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:40:31.376350 18299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:40:31.377996 18299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:40:31.378126 18299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:31.378270 18299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:40:31.378367 18299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:31.414395 18299 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36971
I20260620 23:40:31.414445 18425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36971 every 8 connection(s)
I20260620 23:40:31.416014 18299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
I20260620 23:40:31.422786 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 18299
I20260620 23:40:31.423005 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998525501163-388-0/minicluster-data/ts-1/wal/instance
I20260620 23:40:31.423573 388 external_mini_cluster.cc:1428] Setting key 70998ba2491ae0dc2b6dd22fde4be86d
I20260620 23:40:31.426750 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998525501163-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998525501163-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998525501163-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43365
--builtin_ntp_servers=127.0.97.20:38661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 23:40:31.430552 18297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:31.418771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58997 (local address 127.0.97.62:43365)
0620 23:40:31.419037 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:31.419042 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:31.420684 (+ 1642us) server_negotiation.cc:407] Connection header received
0620 23:40:31.422621 (+ 1937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:31.422625 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:31.422698 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:31.422791 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:40:31.424344 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:31.425200 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:31.427569 (+ 2369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:31.428001 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:31.428105 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:31.429843 (+ 1738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:40:31.429848 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:40:31.429852 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:40:31.429870 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:40:31.429904 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:40:31.429916 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:40:31.429984 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:31.430245 (+ 261us) server_negotiation.cc:299] Negotiation successful
0620 23:40:31.430334 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 23:40:31.432196 18428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:31.419484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43365 (local address 127.0.97.2:58997)
0620 23:40:31.420496 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:31.420553 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 23:40:31.422260 (+ 1707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:40:31.423120 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:40:31.423132 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:40:31.423278 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:40:31.424136 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:31.424163 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:31.425361 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:31.425367 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:31.427417 (+ 2050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:31.427435 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:31.428246 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:31.428255 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:31.428597 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:31.429440 (+ 843us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:40:31.429445 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:40:31.429501 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:40:31.429616 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:40:31.430010 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:40:31.430023 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:40:31.430034 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 23:40:31.430233 (+ 199us) client_negotiation.cc:240] Negotiation successful
0620 23:40:31.430680 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":308,"threads_started":1}
I20260620 23:40:31.433728 18426 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43365
I20260620 23:40:31.434084 18426 heartbeater.cc:464] Registering TS with master...
I20260620 23:40:31.434855 18426 heartbeater.cc:511] Master 127.0.97.62:43365 requested a full tablet report, sending...
I20260620 23:40:31.436501 18109 ts_manager.cc:194] Registered new tserver with Master: be322151ed5f4a15875779fbd9a15693 (127.0.97.2:36971)
I20260620 23:40:31.437353 18109 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:58997
W20260620 23:40:31.781459 18429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:40:31.781956 18429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:40:31.782119 18429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:40:31.791337 18429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:40:31.791465 18429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:40:31.791589 18429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 23:40:31.802026 18429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998525501163-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998525501163-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998525501163-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:40:31.803501 18429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:40:31.805490 18429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:40:31.819782 18434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:40:31.819712 18435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:40:31.819756 18437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:40:31.820675 18429 server_base.cc:1034] running on GCE node
I20260620 23:40:31.821318 18429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:40:31.822484 18429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:40:31.823714 18429 hybrid_clock.cc:648] HybridClock initialized: now 1781998831823660 us; error 49 us; skew 500 ppm
I20260620 23:40:31.824055 18429 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:40:31.826470 18429 webserver.cc:460] Webserver started at http://127.0.97.3:45247/ using document root <none> and password file <none>
I20260620 23:40:31.827283 18429 fs_manager.cc:359] Metadata directory not provided
I20260620 23:40:31.827414 18429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:40:31.827766 18429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:40:31.830842 18429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998525501163-388-0/minicluster-data/ts-2/data/instance:
uuid: "a104bff34b24476dbe25f00ee8844ec3"
format_stamp: "Formatted at 2026-06-20 23:40:31 on dist-test-slave-tcgx"
server_key: "b991cf736e9cfd45d7658fd32b6cad71"
server_key_iv: "20ecb46304fff01e1a321585b287fd18"
server_key_version: "encryptionkey@0"
I20260620 23:40:31.831794 18429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998525501163-388-0/minicluster-data/ts-2/wal/instance:
uuid: "a104bff34b24476dbe25f00ee8844ec3"
format_stamp: "Formatted at 2026-06-20 23:40:31 on dist-test-slave-tcgx"
server_key: "b991cf736e9cfd45d7658fd32b6cad71"
server_key_iv: "20ecb46304fff01e1a321585b287fd18"
server_key_version: "encryptionkey@0"
I20260620 23:40:31.837527 18429 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:40:31.841570 18443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:31.843217 18429 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 23:40:31.843430 18429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998525501163-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998525501163-388-0/minicluster-data/ts-2/wal
uuid: "a104bff34b24476dbe25f00ee8844ec3"
format_stamp: "Formatted at 2026-06-20 23:40:31 on dist-test-slave-tcgx"
server_key: "b991cf736e9cfd45d7658fd32b6cad71"
server_key_iv: "20ecb46304fff01e1a321585b287fd18"
server_key_version: "encryptionkey@0"
I20260620 23:40:31.843701 18429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998525501163-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998525501163-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998525501163-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:40:31.891188 18429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:40:31.892135 18429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:40:31.892444 18429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:40:31.893606 18429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:40:31.895187 18429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:40:31.895323 18429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:31.895462 18429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:40:31.895560 18429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:31.933679 18429 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33625
I20260620 23:40:31.933722 18555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33625 every 8 connection(s)
I20260620 23:40:31.935288 18429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
I20260620 23:40:31.937793 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 18429
I20260620 23:40:31.937989 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998525501163-388-0/minicluster-data/ts-2/wal/instance
I20260620 23:40:31.938660 388 external_mini_cluster.cc:1428] Setting key 93bbe55944b6d76ffd4fa5f90146875b
I20260620 23:40:31.944831 18295 heartbeater.cc:503] Master 127.0.97.62:43365 was elected leader, sending a full tablet report...
I20260620 23:40:31.948367 18558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:31.938029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57743 (local address 127.0.97.62:43365)
0620 23:40:31.938536 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:31.938543 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:31.939984 (+ 1441us) server_negotiation.cc:407] Connection header received
0620 23:40:31.941726 (+ 1742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:31.941731 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:31.941818 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:31.941938 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:40:31.943076 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:31.943807 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:31.945427 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:31.945858 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:31.945986 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:31.947426 (+ 1440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:40:31.947431 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:40:31.947435 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:40:31.947452 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:40:31.947494 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:40:31.947506 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:40:31.947602 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:31.947857 (+ 255us) server_negotiation.cc:299] Negotiation successful
0620 23:40:31.947952 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":230,"threads_started":1}
I20260620 23:40:31.949359 18559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:31.938675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43365 (local address 127.0.97.3:57743)
0620 23:40:31.939809 (+ 1134us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:31.939864 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 23:40:31.941472 (+ 1608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:40:31.942169 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:40:31.942177 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:40:31.942275 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:40:31.942881 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:31.942898 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:31.943950 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:31.943956 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:31.945287 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:31.945299 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:31.946080 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:31.946088 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:31.946309 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:31.947067 (+ 758us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:40:31.947070 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:40:31.947110 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:40:31.947196 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:40:31.947625 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:40:31.947640 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:40:31.947652 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 23:40:31.947823 (+ 171us) client_negotiation.cc:240] Negotiation successful
0620 23:40:31.948243 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":375,"threads_started":1}
I20260620 23:40:31.950644 18556 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43365
I20260620 23:40:31.950973 18556 heartbeater.cc:464] Registering TS with master...
I20260620 23:40:31.951689 18556 heartbeater.cc:511] Master 127.0.97.62:43365 requested a full tablet report, sending...
I20260620 23:40:31.953127 18109 ts_manager.cc:194] Registered new tserver with Master: a104bff34b24476dbe25f00ee8844ec3 (127.0.97.3:33625)
I20260620 23:40:31.953913 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:40:31.954066 18109 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:57743
I20260620 23:40:31.963877 18558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:31.958297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58826 (local address 127.0.97.62:43365)
0620 23:40:31.958525 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:31.958530 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:31.958801 (+ 271us) server_negotiation.cc:407] Connection header received
0620 23:40:31.958995 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:31.959000 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:31.959071 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:31.959152 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:40:31.959629 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:31.960331 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:31.961682 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:31.962067 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:31.962160 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:31.963206 (+ 1046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:40:31.963211 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:40:31.963214 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:40:31.963232 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:40:31.963262 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:40:31.963273 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:40:31.963341 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:31.963462 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 23:40:31.963533 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 23:40:31.967736 18109 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58826:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:40:31.974610 18109 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:40:32.009315 18570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:31.986775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33625 (local address 127.0.0.1:44508)
0620 23:40:31.988390 (+ 1615us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:31.988433 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 23:40:31.988588 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:40:32.001489 (+ 12901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:40:32.001499 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:40:32.001538 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:40:32.002050 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:32.002066 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:32.004294 (+ 2228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:32.004302 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:32.006463 (+ 2161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:32.006475 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:32.008321 (+ 1846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:32.008327 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:32.008526 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:32.008588 (+ 62us) client_negotiation.cc:769] Sending connection context
0620 23:40:32.008727 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 23:40:32.008934 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1357,"mutex_wait_us":38,"thread_start_us":15104,"threads_started":1}
I20260620 23:40:32.009358 18571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:31.987134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44508 (local address 127.0.97.3:33625)
0620 23:40:32.000597 (+ 13463us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:32.000611 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:32.000662 (+ 51us) server_negotiation.cc:407] Connection header received
0620 23:40:32.000809 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:32.000821 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:32.001202 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:32.001452 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:40:32.002464 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:32.004156 (+ 1692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:32.006654 (+ 2498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:32.008129 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:32.008273 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:32.008569 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:32.008792 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 23:40:32.008981 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13206,"thread_start_us":14543,"threads_started":1}
I20260620 23:40:32.010423 18569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:31.986798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36971 (local address 127.0.0.1:46728)
0620 23:40:32.001727 (+ 14929us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:32.001751 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 23:40:32.001883 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:40:32.002698 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:40:32.002702 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:40:32.002729 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:40:32.003196 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:32.003209 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:32.004909 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:32.004917 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:32.006885 (+ 1968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:32.006899 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:32.009720 (+ 2821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:32.009726 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:32.009911 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:32.009944 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 23:40:32.010021 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 23:40:32.010155 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14770,"mutex_wait_us":2,"thread_start_us":1208,"threads_started":1}
I20260620 23:40:32.010732 18573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:31.989435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40587 (local address 127.0.0.1:45006)
0620 23:40:32.001943 (+ 12508us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:32.001963 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:40:32.002092 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:40:32.003253 (+ 1161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:40:32.003257 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:40:32.003282 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:40:32.003683 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:32.003694 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:32.005838 (+ 2144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:32.005844 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:32.007664 (+ 1820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:32.007675 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:32.009885 (+ 2210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:32.009893 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:32.010071 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:32.010104 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 23:40:32.010179 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 23:40:32.010303 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12384,"mutex_wait_us":1,"thread_start_us":12078,"threads_started":1}
I20260620 23:40:32.011242 18572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:31.987765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46728 (local address 127.0.97.2:36971)
0620 23:40:31.989171 (+ 1406us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:31.989181 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:32.002074 (+ 12893us) server_negotiation.cc:407] Connection header received
0620 23:40:32.002194 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:32.002204 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:32.002483 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:32.002685 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:40:32.003407 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:32.004741 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:32.007089 (+ 2348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:32.008533 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:32.008664 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:32.008902 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:32.010711 (+ 1809us) server_negotiation.cc:299] Negotiation successful
0620 23:40:32.010960 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1226,"thread_start_us":1089,"threads_started":1}
I20260620 23:40:32.011626 18574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:31.989544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45006 (local address 127.0.97.1:40587)
0620 23:40:32.002526 (+ 12982us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:32.002537 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:32.002562 (+ 25us) server_negotiation.cc:407] Connection header received
0620 23:40:32.002680 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:32.002687 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:32.002916 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:32.003867 (+ 951us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:40:32.004099 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:32.005680 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:32.007835 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:32.009093 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:32.009216 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:32.011026 (+ 1810us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:32.011115 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 23:40:32.011279 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12816,"thread_start_us":12690,"threads_started":1}
I20260620 23:40:32.013906 18491 tablet_service.cc:1467] Processing CreateTablet for tablet ccd41ca4b9d549a5958f1666ad4bea07 (DEFAULT_TABLE table=test-table [id=d23c4d76876543dfb3a52ce407701f79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:40:32.015215 18491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccd41ca4b9d549a5958f1666ad4bea07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:40:32.014988 18230 tablet_service.cc:1467] Processing CreateTablet for tablet ccd41ca4b9d549a5958f1666ad4bea07 (DEFAULT_TABLE table=test-table [id=d23c4d76876543dfb3a52ce407701f79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:40:32.014988 18361 tablet_service.cc:1467] Processing CreateTablet for tablet ccd41ca4b9d549a5958f1666ad4bea07 (DEFAULT_TABLE table=test-table [id=d23c4d76876543dfb3a52ce407701f79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:40:32.016340 18230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccd41ca4b9d549a5958f1666ad4bea07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:40:32.016348 18361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccd41ca4b9d549a5958f1666ad4bea07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:40:32.027740 18575 tablet_bootstrap.cc:492] T ccd41ca4b9d549a5958f1666ad4bea07 P a104bff34b24476dbe25f00ee8844ec3: Bootstrap starting.
I20260620 23:40:32.031265 18575 tablet_bootstrap.cc:654] T ccd41ca4b9d549a5958f1666ad4bea07 P a104bff34b24476dbe25f00ee8844ec3: Neither blocks nor log segments found. Creating new log.
I20260620 23:40:32.031589 18576 tablet_bootstrap.cc:492] T ccd41ca4b9d549a5958f1666ad4bea07 P be322151ed5f4a15875779fbd9a15693: Bootstrap starting.
I20260620 23:40:32.032864 18575 log.cc:824] T ccd41ca4b9d549a5958f1666ad4bea07 P a104bff34b24476dbe25f00ee8844ec3: Log is configured to *not* fsync() on all Append() calls
I20260620 23:40:32.033210 18577 tablet_bootstrap.cc:492] T ccd41ca4b9d549a5958f1666ad4bea07 P 5663c9cb2afd4218a77060ada53e9ff1: Bootstrap starting.
I20260620 23:40:32.034986 18576 tablet_bootstrap.cc:654] T ccd41ca4b9d549a5958f1666ad4bea07 P be322151ed5f4a15875779fbd9a15693: Neither blocks nor log segments found. Creating new log.
I20260620 23:40:32.036252 18576 log.cc:824] T ccd41ca4b9d549a5958f1666ad4bea07 P be322151ed5f4a15875779fbd9a15693: Log is configured to *not* fsync() on all Append() calls
I20260620 23:40:32.036507 18577 tablet_bootstrap.cc:654] T ccd41ca4b9d549a5958f1666ad4bea07 P 5663c9cb2afd4218a77060ada53e9ff1: Neither blocks nor log segments found. Creating new log.
I20260620 23:40:32.036849 18575 tablet_bootstrap.cc:492] T ccd41ca4b9d549a5958f1666ad4bea07 P a104bff34b24476dbe25f00ee8844ec3: No bootstrap required, opened a new log
I20260620 23:40:32.037577 18575 ts_tablet_manager.cc:1397] T ccd41ca4b9d549a5958f1666ad4bea07 P a104bff34b24476dbe25f00ee8844ec3: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260620 23:40:32.037626 18577 log.cc:824] T ccd41ca4b9d549a5958f1666ad4bea07 P 5663c9cb2afd4218a77060ada53e9ff1: Log is configured to *not* fsync() on all Append() calls
I20260620 23:40:32.040179 18577 tablet_bootstrap.cc:492] T ccd41ca4b9d549a5958f1666ad4bea07 P 5663c9cb2afd4218a77060ada53e9ff1: No bootstrap required, opened a new log
I20260620 23:40:32.040431 18577 ts_tablet_manager.cc:1397] T ccd41ca4b9d549a5958f1666ad4bea07 P 5663c9cb2afd4218a77060ada53e9ff1: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260620 23:40:32.045022 18576 tablet_bootstrap.cc:492] T ccd41ca4b9d549a5958f1666ad4bea07 P be322151ed5f4a15875779fbd9a15693: No bootstrap required, opened a new log
I20260620 23:40:32.045288 18576 ts_tablet_manager.cc:1397] T ccd41ca4b9d549a5958f1666ad4bea07 P be322151ed5f4a15875779fbd9a15693: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.000s
I20260620 23:40:32.045583 18575 raft_consensus.cc:348] T ccd41ca4b9d549a5958f1666ad4bea07 P a104bff34b24476dbe25f00ee8844ec3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a104bff34b24476dbe25f00ee8844ec3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33625 } } peers { permanent_uuid: "be322151ed5f4a15875779fbd9a15693" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36971 } } peers { permanent_uuid: "5663c9cb2afd4218a77060ada53e9ff1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40587 } }
I20260620 23:40:32.046031 18575 raft_consensus.cc:374] T ccd41ca4b9d549a5958f1666ad4bea07 P a104bff34b24476dbe25f00ee8844ec3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:40:32.046139 18575 raft_consensus.cc:729] T ccd41ca4b9d549a5958f1666ad4bea07 P a104bff34b24476dbe25f00ee8844ec3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a104bff34b24476dbe25f00ee8844ec3, State: Initialized, Role: FOLLOWER
I20260620 23:40:32.046765 18575 consensus_queue.cc:260] T ccd41ca4b9d549a5958f1666ad4bea07 P a104bff34b24476dbe25f00ee8844ec3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a104bff34b24476dbe25f00ee8844ec3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33625 } } peers { permanent_uuid: "be322151ed5f4a15875779fbd9a15693" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36971 } } peers { permanent_uuid: "5663c9cb2afd4218a77060ada53e9ff1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40587 } }
I20260620 23:40:32.047977 18556 heartbeater.cc:503] Master 127.0.97.62:43365 was elected leader, sending a full tablet report...
I20260620 23:40:32.048563 18575 ts_tablet_manager.cc:1428] T ccd41ca4b9d549a5958f1666ad4bea07 P a104bff34b24476dbe25f00ee8844ec3: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260620 23:40:32.047742 18577 raft_consensus.cc:348] T ccd41ca4b9d549a5958f1666ad4bea07 P 5663c9cb2afd4218a77060ada53e9ff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a104bff34b24476dbe25f00ee8844ec3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33625 } } peers { permanent_uuid: "be322151ed5f4a15875779fbd9a15693" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36971 } } peers { permanent_uuid: "5663c9cb2afd4218a77060ada53e9ff1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40587 } }
I20260620 23:40:32.049209 18577 raft_consensus.cc:374] T ccd41ca4b9d549a5958f1666ad4bea07 P 5663c9cb2afd4218a77060ada53e9ff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:40:32.049306 18577 raft_consensus.cc:729] T ccd41ca4b9d549a5958f1666ad4bea07 P 5663c9cb2afd4218a77060ada53e9ff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5663c9cb2afd4218a77060ada53e9ff1, State: Initialized, Role: FOLLOWER
I20260620 23:40:32.049923 18577 consensus_queue.cc:260] T ccd41ca4b9d549a5958f1666ad4bea07 P 5663c9cb2afd4218a77060ada53e9ff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a104bff34b24476dbe25f00ee8844ec3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33625 } } peers { permanent_uuid: "be322151ed5f4a15875779fbd9a15693" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36971 } } peers { permanent_uuid: "5663c9cb2afd4218a77060ada53e9ff1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40587 } }
I20260620 23:40:32.051663 18577 ts_tablet_manager.cc:1428] T ccd41ca4b9d549a5958f1666ad4bea07 P 5663c9cb2afd4218a77060ada53e9ff1: Time spent starting tablet: real 0.011s user 0.008s sys 0.003s
I20260620 23:40:32.056888 18576 raft_consensus.cc:348] T ccd41ca4b9d549a5958f1666ad4bea07 P be322151ed5f4a15875779fbd9a15693 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a104bff34b24476dbe25f00ee8844ec3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33625 } } peers { permanent_uuid: "be322151ed5f4a15875779fbd9a15693" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36971 } } peers { permanent_uuid: "5663c9cb2afd4218a77060ada53e9ff1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40587 } }
I20260620 23:40:32.057240 18576 raft_consensus.cc:374] T ccd41ca4b9d549a5958f1666ad4bea07 P be322151ed5f4a15875779fbd9a15693 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:40:32.057351 18576 raft_consensus.cc:729] T ccd41ca4b9d549a5958f1666ad4bea07 P be322151ed5f4a15875779fbd9a15693 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be322151ed5f4a15875779fbd9a15693, State: Initialized, Role: FOLLOWER
I20260620 23:40:32.057917 18576 consensus_queue.cc:260] T ccd41ca4b9d549a5958f1666ad4bea07 P be322151ed5f4a15875779fbd9a15693 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a104bff34b24476dbe25f00ee8844ec3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33625 } } peers { permanent_uuid: "be322151ed5f4a15875779fbd9a15693" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36971 } } peers { permanent_uuid: "5663c9cb2afd4218a77060ada53e9ff1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40587 } }
I20260620 23:40:32.058862 18426 heartbeater.cc:503] Master 127.0.97.62:43365 was elected leader, sending a full tablet report...
I20260620 23:40:32.059331 18576 ts_tablet_manager.cc:1428] T ccd41ca4b9d549a5958f1666ad4bea07 P be322151ed5f4a15875779fbd9a15693: Time spent starting tablet: real 0.014s user 0.012s sys 0.001s
W20260620 23:40:32.168288 18427 tablet.cc:2369] T ccd41ca4b9d549a5958f1666ad4bea07 P be322151ed5f4a15875779fbd9a15693: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:40:32.170295 18296 tablet.cc:2369] T ccd41ca4b9d549a5958f1666ad4bea07 P 5663c9cb2afd4218a77060ada53e9ff1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:40:32.187137 18557 tablet.cc:2369] T ccd41ca4b9d549a5958f1666ad4bea07 P a104bff34b24476dbe25f00ee8844ec3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:40:32.196905 18582 raft_consensus.cc:482] T ccd41ca4b9d549a5958f1666ad4bea07 P 5663c9cb2afd4218a77060ada53e9ff1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:40:32.197261 18582 raft_consensus.cc:504] T ccd41ca4b9d549a5958f1666ad4bea07 P 5663c9cb2afd4218a77060ada53e9ff1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a104bff34b24476dbe25f00ee8844ec3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33625 } } peers { permanent_uuid: "be322151ed5f4a15875779fbd9a15693" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36971 } } peers { permanent_uuid: "5663c9cb2afd4218a77060ada53e9ff1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40587 } }
I20260620 23:40:32.199003 18582 leader_election.cc:288] T ccd41ca4b9d549a5958f1666ad4bea07 P 5663c9cb2afd4218a77060ada53e9ff1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a104bff34b24476dbe25f00ee8844ec3 (127.0.97.3:33625), be322151ed5f4a15875779fbd9a15693 (127.0.97.2:36971)
I20260620 23:40:32.205791 18571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:32.199424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35783 (local address 127.0.97.3:33625)
0620 23:40:32.199645 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:32.199652 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:32.200515 (+ 863us) server_negotiation.cc:407] Connection header received
0620 23:40:32.200687 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:32.200694 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:32.200770 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:32.200880 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:40:32.201464 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:32.202183 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:32.204147 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:32.204900 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:32.205075 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:32.205175 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:32.205509 (+ 334us) server_negotiation.cc:299] Negotiation successful
0620 23:40:32.205619 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 23:40:32.205793 18572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:32.199563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38935 (local address 127.0.97.2:36971)
0620 23:40:32.199796 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:32.199805 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:32.200326 (+ 521us) server_negotiation.cc:407] Connection header received
0620 23:40:32.200513 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:32.200519 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:32.200611 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:32.200693 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:40:32.201327 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:32.202076 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:32.204117 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:32.204864 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:32.204961 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:32.205072 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:32.205419 (+ 347us) server_negotiation.cc:299] Negotiation successful
0620 23:40:32.205541 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 23:40:32.205876 18584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:32.199325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36971 (local address 127.0.97.1:38935)
0620 23:40:32.200216 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:32.200232 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:40:32.200371 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:40:32.200736 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:40:32.200741 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:40:32.200764 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:40:32.201177 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:32.201186 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:32.202213 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:32.202221 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:32.203931 (+ 1710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:32.203942 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:32.205049 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:32.205055 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:32.205251 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:32.205275 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 23:40:32.205391 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 23:40:32.205520 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":778,"mutex_wait_us":87,"thread_start_us":523,"threads_started":1}
I20260620 23:40:32.206488 18585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:32.199383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33625 (local address 127.0.97.1:35783)
0620 23:40:32.200366 (+ 983us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:32.200384 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:40:32.200531 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:40:32.200894 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:40:32.200897 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:40:32.200928 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:40:32.201312 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:32.201322 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:32.202311 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:32.202314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:32.203946 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:32.203958 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:32.205121 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:32.205127 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:32.205331 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:32.205361 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 23:40:32.206072 (+ 711us) client_negotiation.cc:240] Negotiation successful
0620 23:40:32.206184 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":845,"mutex_wait_us":33,"thread_start_us":520,"threads_started":1}
I20260620 23:40:32.207024 18381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccd41ca4b9d549a5958f1666ad4bea07" candidate_uuid: "5663c9cb2afd4218a77060ada53e9ff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be322151ed5f4a15875779fbd9a15693" is_pre_election: true
I20260620 23:40:32.207515 18581 raft_consensus.cc:482] T ccd41ca4b9d549a5958f1666ad4bea07 P a104bff34b24476dbe25f00ee8844ec3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:40:32.207588 18381 raft_consensus.cc:2454] T ccd41ca4b9d549a5958f1666ad4bea07 P be322151ed5f4a15875779fbd9a15693 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5663c9cb2afd4218a77060ada53e9ff1 in term 0.
I20260620 23:40:32.207644 18511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccd41ca4b9d549a5958f1666ad4bea07" candidate_uuid: "5663c9cb2afd4218a77060ada53e9ff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a104bff34b24476dbe25f00ee8844ec3" is_pre_election: true
I20260620 23:40:32.207849 18581 raft_consensus.cc:504] T ccd41ca4b9d549a5958f1666ad4bea07 P a104bff34b24476dbe25f00ee8844ec3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a104bff34b24476dbe25f00ee8844ec3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33625 } } peers { permanent_uuid: "be322151ed5f4a15875779fbd9a15693" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36971 } } peers { permanent_uuid: "5663c9cb2afd4218a77060ada53e9ff1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40587 } }
I20260620 23:40:32.208278 18186 leader_election.cc:302] T ccd41ca4b9d549a5958f1666ad4bea07 P 5663c9cb2afd4218a77060ada53e9ff1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5663c9cb2afd4218a77060ada53e9ff1, be322151ed5f4a15875779fbd9a15693; no voters:
I20260620 23:40:32.208585 18511 raft_consensus.cc:2454] T ccd41ca4b9d549a5958f1666ad4bea07 P a104bff34b24476dbe25f00ee8844ec3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5663c9cb2afd4218a77060ada53e9ff1 in term 0.
I20260620 23:40:32.208693 18582 raft_consensus.cc:2781] T ccd41ca4b9d549a5958f1666ad4bea07 P 5663c9cb2afd4218a77060ada53e9ff1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:40:32.208868 18582 raft_consensus.cc:482] T ccd41ca4b9d549a5958f1666ad4bea07 P 5663c9cb2afd4218a77060ada53e9ff1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:40:32.208982 18582 raft_consensus.cc:3037] T ccd41ca4b9d549a5958f1666ad4bea07 P 5663c9cb2afd4218a77060ada53e9ff1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:40:32.209868 18581 leader_election.cc:288] T ccd41ca4b9d549a5958f1666ad4bea07 P a104bff34b24476dbe25f00ee8844ec3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be322151ed5f4a15875779fbd9a15693 (127.0.97.2:36971), 5663c9cb2afd4218a77060ada53e9ff1 (127.0.97.1:40587)
I20260620 23:40:32.212037 18582 raft_consensus.cc:504] T ccd41ca4b9d549a5958f1666ad4bea07 P 5663c9cb2afd4218a77060ada53e9ff1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a104bff34b24476dbe25f00ee8844ec3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33625 } } peers { permanent_uuid: "be322151ed5f4a15875779fbd9a15693" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36971 } } peers { permanent_uuid: "5663c9cb2afd4218a77060ada53e9ff1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40587 } }
I20260620 23:40:32.212658 18582 leader_election.cc:288] T ccd41ca4b9d549a5958f1666ad4bea07 P 5663c9cb2afd4218a77060ada53e9ff1 [CANDIDATE]: Term 1 election: Requested vote from peers a104bff34b24476dbe25f00ee8844ec3 (127.0.97.3:33625), be322151ed5f4a15875779fbd9a15693 (127.0.97.2:36971)
I20260620 23:40:32.213109 18511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccd41ca4b9d549a5958f1666ad4bea07" candidate_uuid: "5663c9cb2afd4218a77060ada53e9ff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a104bff34b24476dbe25f00ee8844ec3"
I20260620 23:40:32.213379 18511 raft_consensus.cc:3037] T ccd41ca4b9d549a5958f1666ad4bea07 P a104bff34b24476dbe25f00ee8844ec3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:40:32.213299 18381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccd41ca4b9d549a5958f1666ad4bea07" candidate_uuid: "5663c9cb2afd4218a77060ada53e9ff1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be322151ed5f4a15875779fbd9a15693"
I20260620 23:40:32.213723 18381 raft_consensus.cc:3037] T ccd41ca4b9d549a5958f1666ad4bea07 P be322151ed5f4a15875779fbd9a15693 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:40:32.217027 18381 raft_consensus.cc:2454] T ccd41ca4b9d549a5958f1666ad4bea07 P be322151ed5f4a15875779fbd9a15693 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5663c9cb2afd4218a77060ada53e9ff1 in term 1.
I20260620 23:40:32.217458 18511 raft_consensus.cc:2454] T ccd41ca4b9d549a5958f1666ad4bea07 P a104bff34b24476dbe25f00ee8844ec3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5663c9cb2afd4218a77060ada53e9ff1 in term 1.
I20260620 23:40:32.217463 18186 leader_election.cc:302] T ccd41ca4b9d549a5958f1666ad4bea07 P 5663c9cb2afd4218a77060ada53e9ff1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5663c9cb2afd4218a77060ada53e9ff1, be322151ed5f4a15875779fbd9a15693; no voters:
I20260620 23:40:32.217926 18559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:32.210066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36971 (local address 127.0.97.3:41369)
0620 23:40:32.210252 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:32.210273 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:40:32.210394 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:40:32.210969 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:40:32.210974 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:40:32.211010 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:40:32.211194 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:32.211206 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:32.213031 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:32.213040 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:32.215379 (+ 2339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:32.215393 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:32.217093 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:32.217099 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:32.217332 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:32.217371 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 23:40:32.217435 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 23:40:32.217585 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 23:40:32.218204 18582 raft_consensus.cc:2781] T ccd41ca4b9d549a5958f1666ad4bea07 P 5663c9cb2afd4218a77060ada53e9ff1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:40:32.218844 18572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:32.210412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41369 (local address 127.0.97.2:36971)
0620 23:40:32.210586 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:32.210591 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:32.210610 (+ 19us) server_negotiation.cc:407] Connection header received
0620 23:40:32.210685 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:32.210693 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:32.210803 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:32.210954 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:40:32.211360 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:32.212825 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:32.215587 (+ 2762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:32.216897 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:32.218251 (+ 1354us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:32.218348 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:32.218400 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 23:40:32.218545 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 23:40:32.219496 18380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccd41ca4b9d549a5958f1666ad4bea07" candidate_uuid: "a104bff34b24476dbe25f00ee8844ec3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be322151ed5f4a15875779fbd9a15693" is_pre_election: true
I20260620 23:40:32.219903 18380 raft_consensus.cc:2379] T ccd41ca4b9d549a5958f1666ad4bea07 P be322151ed5f4a15875779fbd9a15693 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a104bff34b24476dbe25f00ee8844ec3 in current term 1: Already voted for candidate 5663c9cb2afd4218a77060ada53e9ff1 in this term.
I20260620 23:40:32.220703 18574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:32.210658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57539 (local address 127.0.97.1:40587)
0620 23:40:32.210802 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:32.210808 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:32.213696 (+ 2888us) server_negotiation.cc:407] Connection header received
0620 23:40:32.213980 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:32.213986 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:32.214075 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:32.214136 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:40:32.215002 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:32.215987 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:32.218534 (+ 2547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:32.219630 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:32.219764 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:32.219872 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:32.220310 (+ 438us) server_negotiation.cc:299] Negotiation successful
0620 23:40:32.220464 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 23:40:32.220890 18586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:32.210177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40587 (local address 127.0.97.3:57539)
0620 23:40:32.213596 (+ 3419us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:32.213615 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:40:32.213803 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:40:32.214309 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:40:32.214313 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:40:32.214335 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:40:32.214786 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:32.214797 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:32.216177 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:32.216183 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:32.218359 (+ 2176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:32.218378 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:32.219826 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:32.219844 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:32.220097 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:32.220152 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 23:40:32.220267 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 23:40:32.220472 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3288,"thread_start_us":8327,"threads_started":1}
I20260620 23:40:32.221644 18250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccd41ca4b9d549a5958f1666ad4bea07" candidate_uuid: "a104bff34b24476dbe25f00ee8844ec3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5663c9cb2afd4218a77060ada53e9ff1" is_pre_election: true
I20260620 23:40:32.226598 18582 raft_consensus.cc:686] T ccd41ca4b9d549a5958f1666ad4bea07 P 5663c9cb2afd4218a77060ada53e9ff1 [term 1 LEADER]: Becoming Leader. State: Replica: 5663c9cb2afd4218a77060ada53e9ff1, State: Running, Role: LEADER
I20260620 23:40:32.227177 18582 consensus_queue.cc:237] T ccd41ca4b9d549a5958f1666ad4bea07 P 5663c9cb2afd4218a77060ada53e9ff1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a104bff34b24476dbe25f00ee8844ec3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33625 } } peers { permanent_uuid: "be322151ed5f4a15875779fbd9a15693" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36971 } } peers { permanent_uuid: "5663c9cb2afd4218a77060ada53e9ff1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40587 } }
I20260620 23:40:32.231503 18446 leader_election.cc:302] T ccd41ca4b9d549a5958f1666ad4bea07 P a104bff34b24476dbe25f00ee8844ec3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a104bff34b24476dbe25f00ee8844ec3; no voters: 5663c9cb2afd4218a77060ada53e9ff1, be322151ed5f4a15875779fbd9a15693
I20260620 23:40:32.231976 18581 raft_consensus.cc:2726] T ccd41ca4b9d549a5958f1666ad4bea07 P a104bff34b24476dbe25f00ee8844ec3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 23:40:32.232471 18109 catalog_manager.cc:5495] T ccd41ca4b9d549a5958f1666ad4bea07 P 5663c9cb2afd4218a77060ada53e9ff1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5663c9cb2afd4218a77060ada53e9ff1 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "5663c9cb2afd4218a77060ada53e9ff1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a104bff34b24476dbe25f00ee8844ec3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be322151ed5f4a15875779fbd9a15693" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5663c9cb2afd4218a77060ada53e9ff1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40587 } health_report { overall_health: HEALTHY } } }
I20260620 23:40:32.292338 18574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:32.287020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45012 (local address 127.0.97.1:40587)
0620 23:40:32.287192 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:32.287197 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:32.287215 (+ 18us) server_negotiation.cc:407] Connection header received
0620 23:40:32.287305 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:32.287309 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:32.287432 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:32.287556 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:40:32.287858 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:32.288622 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:32.290158 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:32.290585 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:32.290676 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:32.291225 (+ 549us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:40:32.291756 (+ 531us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:40:32.291877 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:32.291937 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 23:40:32.292001 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 23:40:32.300165 18381 raft_consensus.cc:1261] T ccd41ca4b9d549a5958f1666ad4bea07 P be322151ed5f4a15875779fbd9a15693 [term 1 FOLLOWER]: Refusing update from remote peer 5663c9cb2afd4218a77060ada53e9ff1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:40:32.300235 18511 raft_consensus.cc:1261] T ccd41ca4b9d549a5958f1666ad4bea07 P a104bff34b24476dbe25f00ee8844ec3 [term 1 FOLLOWER]: Refusing update from remote peer 5663c9cb2afd4218a77060ada53e9ff1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:40:32.301028 18587 consensus_queue.cc:1035] T ccd41ca4b9d549a5958f1666ad4bea07 P 5663c9cb2afd4218a77060ada53e9ff1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a104bff34b24476dbe25f00ee8844ec3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:40:32.301390 18582 consensus_queue.cc:1035] T ccd41ca4b9d549a5958f1666ad4bea07 P 5663c9cb2afd4218a77060ada53e9ff1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be322151ed5f4a15875779fbd9a15693" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:40:32.313059 18590 mvcc.cc:205] Tried to move back new op lower bound from 7299067217092403200 to 7299067216806477824. Current Snapshot: MvccSnapshot[applied={T|T < 7299067217092403200}]
I20260620 23:40:32.328364 18109 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:40:32.328661 18109 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:40:32.332397 18109 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fc66e800bb0f44f9af07d6ee58230318: Sending DeleteTablet for 3 replicas of tablet ccd41ca4b9d549a5958f1666ad4bea07
I20260620 23:40:32.333611 18491 tablet_service.cc:1514] Processing DeleteTablet for tablet ccd41ca4b9d549a5958f1666ad4bea07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:40:32 UTC) from {username='slave'} at 127.0.0.1:44508
I20260620 23:40:32.333655 18361 tablet_service.cc:1514] Processing DeleteTablet for tablet ccd41ca4b9d549a5958f1666ad4bea07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:40:32 UTC) from {username='slave'} at 127.0.0.1:46728
I20260620 23:40:32.333658 18230 tablet_service.cc:1514] Processing DeleteTablet for tablet ccd41ca4b9d549a5958f1666ad4bea07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:40:32 UTC) from {username='slave'} at 127.0.0.1:45006
I20260620 23:40:32.335533 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 18168
I20260620 23:40:32.337126 18599 tablet_replica.cc:330] T ccd41ca4b9d549a5958f1666ad4bea07 P a104bff34b24476dbe25f00ee8844ec3: stopping tablet replica
I20260620 23:40:32.337579 18599 raft_consensus.cc:2229] T ccd41ca4b9d549a5958f1666ad4bea07 P a104bff34b24476dbe25f00ee8844ec3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:40:32.338081 18599 raft_consensus.cc:2258] T ccd41ca4b9d549a5958f1666ad4bea07 P a104bff34b24476dbe25f00ee8844ec3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:40:32.338526 18601 tablet_replica.cc:330] T ccd41ca4b9d549a5958f1666ad4bea07 P be322151ed5f4a15875779fbd9a15693: stopping tablet replica
I20260620 23:40:32.338979 18601 raft_consensus.cc:2229] T ccd41ca4b9d549a5958f1666ad4bea07 P be322151ed5f4a15875779fbd9a15693 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:40:32.339432 18601 raft_consensus.cc:2258] T ccd41ca4b9d549a5958f1666ad4bea07 P be322151ed5f4a15875779fbd9a15693 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:40:32.341315 18601 ts_tablet_manager.cc:1905] T ccd41ca4b9d549a5958f1666ad4bea07 P be322151ed5f4a15875779fbd9a15693: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:40:32.349290 18600 tablet_replica.cc:330] T ccd41ca4b9d549a5958f1666ad4bea07 P 5663c9cb2afd4218a77060ada53e9ff1: stopping tablet replica
I20260620 23:40:32.349751 18600 raft_consensus.cc:2229] T ccd41ca4b9d549a5958f1666ad4bea07 P 5663c9cb2afd4218a77060ada53e9ff1 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:40:32.350437 18600 raft_consensus.cc:2258] T ccd41ca4b9d549a5958f1666ad4bea07 P 5663c9cb2afd4218a77060ada53e9ff1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:40:32.351089 18599 ts_tablet_manager.cc:1905] T ccd41ca4b9d549a5958f1666ad4bea07 P a104bff34b24476dbe25f00ee8844ec3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:40:32.351408 18601 ts_tablet_manager.cc:1918] T ccd41ca4b9d549a5958f1666ad4bea07 P be322151ed5f4a15875779fbd9a15693: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:40:32.351567 18601 log.cc:1196] T ccd41ca4b9d549a5958f1666ad4bea07 P be322151ed5f4a15875779fbd9a15693: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998525501163-388-0/minicluster-data/ts-1/wal/wals/ccd41ca4b9d549a5958f1666ad4bea07
I20260620 23:40:32.352254 18601 ts_tablet_manager.cc:1939] T ccd41ca4b9d549a5958f1666ad4bea07 P be322151ed5f4a15875779fbd9a15693: Deleting consensus metadata
I20260620 23:40:32.354215 18096 catalog_manager.cc:4841] TS be322151ed5f4a15875779fbd9a15693 (127.0.97.2:36971): tablet ccd41ca4b9d549a5958f1666ad4bea07 (table test-table [id=d23c4d76876543dfb3a52ce407701f79]) successfully deleted
I20260620 23:40:32.358516 18599 ts_tablet_manager.cc:1918] T ccd41ca4b9d549a5958f1666ad4bea07 P a104bff34b24476dbe25f00ee8844ec3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:40:32.358686 18599 log.cc:1196] T ccd41ca4b9d549a5958f1666ad4bea07 P a104bff34b24476dbe25f00ee8844ec3: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998525501163-388-0/minicluster-data/ts-2/wal/wals/ccd41ca4b9d549a5958f1666ad4bea07
I20260620 23:40:32.359194 18599 ts_tablet_manager.cc:1939] T ccd41ca4b9d549a5958f1666ad4bea07 P a104bff34b24476dbe25f00ee8844ec3: Deleting consensus metadata
I20260620 23:40:32.360829 18600 ts_tablet_manager.cc:1905] T ccd41ca4b9d549a5958f1666ad4bea07 P 5663c9cb2afd4218a77060ada53e9ff1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:40:32.360982 18094 catalog_manager.cc:4841] TS a104bff34b24476dbe25f00ee8844ec3 (127.0.97.3:33625): tablet ccd41ca4b9d549a5958f1666ad4bea07 (table test-table [id=d23c4d76876543dfb3a52ce407701f79]) successfully deleted
I20260620 23:40:32.365319 18574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:32.336538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45026 (local address 127.0.97.1:40587)
0620 23:40:32.336754 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:32.336764 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:32.349731 (+ 12967us) server_negotiation.cc:407] Connection header received
0620 23:40:32.350036 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:32.350042 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:32.350197 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:32.350306 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:40:32.351061 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:32.352398 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:32.355219 (+ 2821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:32.360244 (+ 5025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:32.360383 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:32.363502 (+ 3119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:40:32.363513 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:40:32.363530 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:40:32.364376 (+ 846us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:40:32.364463 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:40:32.364489 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:40:32.364585 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:32.364855 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 23:40:32.364977 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 23:40:32.366474 18290 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:40:32.462507 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 18168
W20260620 23:40:32.480513 18095 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:40587 (error 108)
I20260620 23:40:32.481272 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 18299
I20260620 23:40:32.482254 18573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:32.481766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40587 (local address 127.0.0.1:45032)
0620 23:40:32.481945 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:32.482075 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 23:40:32.482506 18095 catalog_manager.cc:4584] TS 5663c9cb2afd4218a77060ada53e9ff1 (127.0.97.1:40587): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ccd41ca4b9d549a5958f1666ad4bea07: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40587: connect: Connection refused (error 111)
I20260620 23:40:32.488013 18572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:32.481958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46732 (local address 127.0.97.2:36971)
0620 23:40:32.482123 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:32.482127 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:32.482178 (+ 51us) server_negotiation.cc:407] Connection header received
0620 23:40:32.482408 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:32.482413 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:32.482533 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:32.482633 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:40:32.483120 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:32.483888 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:32.485142 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:32.485515 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:32.485615 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:32.486612 (+ 997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:40:32.486619 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:40:32.486633 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:40:32.487312 (+ 679us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:40:32.487363 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:40:32.487381 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:40:32.487471 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:32.487610 (+ 139us) server_negotiation.cc:299] Negotiation successful
0620 23:40:32.487695 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 23:40:32.488656 18421 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:40:32.536209 18573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:32.535747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40587 (local address 127.0.0.1:45034)
0620 23:40:32.535971 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:32.536058 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 23:40:32.537169 18573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:32.536843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40587 (local address 127.0.0.1:45048)
0620 23:40:32.536984 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:32.537053 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 23:40:32.579423 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 18299
I20260620 23:40:32.597061 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 18429
I20260620 23:40:32.597864 18573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:32.597302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40587 (local address 127.0.0.1:45062)
0620 23:40:32.597524 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:32.597636 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260620 23:40:32.598839 18573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:32.598517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40587 (local address 127.0.0.1:45078)
0620 23:40:32.598662 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:32.598732 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 23:40:32.603957 18571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:32.597845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44512 (local address 127.0.97.3:33625)
0620 23:40:32.598032 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:32.598039 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:32.598061 (+ 22us) server_negotiation.cc:407] Connection header received
0620 23:40:32.598224 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:32.598230 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:32.598330 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:32.598449 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:40:32.598856 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:32.599639 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:32.600975 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:32.601378 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:32.601508 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:32.602515 (+ 1007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:40:32.602526 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:40:32.602541 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:40:32.603212 (+ 671us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:40:32.603260 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:40:32.603280 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:40:32.603408 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:32.603531 (+ 123us) server_negotiation.cc:299] Negotiation successful
0620 23:40:32.603637 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 23:40:32.604595 18551 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:40:32.679322 18573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:32.678810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40587 (local address 127.0.0.1:45082)
0620 23:40:32.679045 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:32.679139 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 23:40:32.680429 18573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:32.679980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40587 (local address 127.0.0.1:45090)
0620 23:40:32.680200 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:32.680277 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260620 23:40:32.696521 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 18429
I20260620 23:40:32.713354 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 18078
I20260620 23:40:32.714504 18139 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:40:32.824009 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 18078
2026-06-20T23:40:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3017 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:40:32 dist-test-slave-tcgx krb5kdc[18610](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:40:32 dist-test-slave-tcgx krb5kdc[18610](info): set up 2 sockets
Jun 20 23:40:32 dist-test-slave-tcgx krb5kdc[18610](info): commencing operation
krb5kdc: starting...
W20260620 23:40:33.955065 388 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/krb5kdc/test-user.keytab.
Jun 20 23:40:34 dist-test-slave-tcgx krb5kdc[18610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:40:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:40:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 23:40:34.229871 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34577
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34577
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:40:34.568590 18623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:40:34.568913 18623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:40:34.569005 18623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:40:34.576412 18623 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:40:34.576530 18623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:40:34.576581 18623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:40:34.576630 18623 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:40:34.576666 18623 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:40:34.586642 18623 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34577
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:40:34.588091 18623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:40:34.589358 18623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:40:34.601796 18628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:40:34.601958 18629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:40:34.602443 18631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:40:34.602771 18623 server_base.cc:1034] running on GCE node
I20260620 23:40:34.603653 18623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:40:34.605041 18623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:40:34.606240 18623 hybrid_clock.cc:648] HybridClock initialized: now 1781998834606182 us; error 57 us; skew 500 ppm
I20260620 23:40:34.606598 18623 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:40:34 dist-test-slave-tcgx krb5kdc[18610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998834, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:40:34.609819 18623 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 23:40:34.611408 18623 webserver.cc:460] Webserver started at http://127.0.97.62:36775/ using document root <none> and password file <none>
I20260620 23:40:34.612253 18623 fs_manager.cc:359] Metadata directory not provided
I20260620 23:40:34.612363 18623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:40:34.612632 18623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:40:34.615787 18623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/minicluster-data/master-0/data/instance:
uuid: "3c71d4c8ad3541049653ecace2c173db"
format_stamp: "Formatted at 2026-06-20 23:40:34 on dist-test-slave-tcgx"
server_key: "c899da0ba2dcc06a56519059a279bc25"
server_key_iv: "da891a39218a866137502101feba5f14"
server_key_version: "encryptionkey@0"
I20260620 23:40:34.616482 18623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/minicluster-data/master-0/wal/instance:
uuid: "3c71d4c8ad3541049653ecace2c173db"
format_stamp: "Formatted at 2026-06-20 23:40:34 on dist-test-slave-tcgx"
server_key: "c899da0ba2dcc06a56519059a279bc25"
server_key_iv: "da891a39218a866137502101feba5f14"
server_key_version: "encryptionkey@0"
I20260620 23:40:34.621709 18623 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260620 23:40:34.625221 18638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:34.626606 18623 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 23:40:34.626842 18623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/minicluster-data/master-0/wal
uuid: "3c71d4c8ad3541049653ecace2c173db"
format_stamp: "Formatted at 2026-06-20 23:40:34 on dist-test-slave-tcgx"
server_key: "c899da0ba2dcc06a56519059a279bc25"
server_key_iv: "da891a39218a866137502101feba5f14"
server_key_version: "encryptionkey@0"
I20260620 23:40:34.627130 18623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:40:34.655931 18623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:40:34.660439 18623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:40:34.660753 18623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:40:34.680838 18623 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34577
I20260620 23:40:34.680876 18689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34577 every 8 connection(s)
I20260620 23:40:34.682430 18623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:40:34.685756 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 18623
I20260620 23:40:34.685979 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/minicluster-data/master-0/wal/instance
I20260620 23:40:34.686510 388 external_mini_cluster.cc:1428] Setting key e2b3f02188f6ea407c7bba738853960f
I20260620 23:40:34.687597 18690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:40:34.700186 18690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c71d4c8ad3541049653ecace2c173db: Bootstrap starting.
Jun 20 23:40:34 dist-test-slave-tcgx krb5kdc[18610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:40:34.704062 18690 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c71d4c8ad3541049653ecace2c173db: Neither blocks nor log segments found. Creating new log.
I20260620 23:40:34.705220 18690 log.cc:824] T 00000000000000000000000000000000 P 3c71d4c8ad3541049653ecace2c173db: Log is configured to *not* fsync() on all Append() calls
I20260620 23:40:34.708479 18690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c71d4c8ad3541049653ecace2c173db: No bootstrap required, opened a new log
I20260620 23:40:34.709436 18693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:34.688598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46006 (local address 127.0.97.62:34577)
0620 23:40:34.689396 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:34.689409 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:34.689450 (+ 41us) server_negotiation.cc:407] Connection header received
0620 23:40:34.691154 (+ 1704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:34.691179 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:34.691615 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:34.692077 (+ 462us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:40:34.693620 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:34.695048 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:34.696440 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:34.697200 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:34.697416 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:34.701684 (+ 4268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:40:34.701693 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:40:34.701711 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:40:34.701762 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:40:34.705037 (+ 3275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:40:34.705904 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:40:34.705912 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:40:34.705919 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:40:34.706107 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:40:34.706514 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:40:34.706520 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:40:34.706525 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:40:34.707159 (+ 634us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:40:34.707447 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:34.707945 (+ 498us) server_negotiation.cc:299] Negotiation successful
0620 23:40:34.708368 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":345,"threads_started":1}
I20260620 23:40:34.717029 18690 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c71d4c8ad3541049653ecace2c173db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c71d4c8ad3541049653ecace2c173db" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34577 } }
I20260620 23:40:34.717548 18690 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c71d4c8ad3541049653ecace2c173db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:40:34.717679 18690 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c71d4c8ad3541049653ecace2c173db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c71d4c8ad3541049653ecace2c173db, State: Initialized, Role: FOLLOWER
I20260620 23:40:34.718250 18690 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c71d4c8ad3541049653ecace2c173db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c71d4c8ad3541049653ecace2c173db" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34577 } }
I20260620 23:40:34.718519 18690 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c71d4c8ad3541049653ecace2c173db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:40:34.718677 18690 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c71d4c8ad3541049653ecace2c173db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:40:34.718860 18690 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c71d4c8ad3541049653ecace2c173db [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:40:34.721215 18690 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c71d4c8ad3541049653ecace2c173db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c71d4c8ad3541049653ecace2c173db" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34577 } }
I20260620 23:40:34.721766 18690 leader_election.cc:302] T 00000000000000000000000000000000 P 3c71d4c8ad3541049653ecace2c173db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c71d4c8ad3541049653ecace2c173db; no voters:
I20260620 23:40:34.722402 18690 leader_election.cc:288] T 00000000000000000000000000000000 P 3c71d4c8ad3541049653ecace2c173db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:40:34.722544 18695 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c71d4c8ad3541049653ecace2c173db [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:40:34.723579 18695 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c71d4c8ad3541049653ecace2c173db [term 1 LEADER]: Becoming Leader. State: Replica: 3c71d4c8ad3541049653ecace2c173db, State: Running, Role: LEADER
I20260620 23:40:34.724447 18695 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c71d4c8ad3541049653ecace2c173db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c71d4c8ad3541049653ecace2c173db" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34577 } }
I20260620 23:40:34.724885 18690 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c71d4c8ad3541049653ecace2c173db [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:40:34.727180 18696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c71d4c8ad3541049653ecace2c173db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c71d4c8ad3541049653ecace2c173db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c71d4c8ad3541049653ecace2c173db" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34577 } } }
I20260620 23:40:34.727702 18697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c71d4c8ad3541049653ecace2c173db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c71d4c8ad3541049653ecace2c173db. Latest consensus state: current_term: 1 leader_uuid: "3c71d4c8ad3541049653ecace2c173db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c71d4c8ad3541049653ecace2c173db" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34577 } } }
I20260620 23:40:34.727716 18696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c71d4c8ad3541049653ecace2c173db [sys.catalog]: This master's current role is: LEADER
I20260620 23:40:34.728117 18697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c71d4c8ad3541049653ecace2c173db [sys.catalog]: This master's current role is: LEADER
I20260620 23:40:34.730290 18703 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:40:34.735885 18703 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:40:34.745015 18703 catalog_manager.cc:1347] Generated new cluster ID: e9b31199eefb4f93922c131d69e0d33b
I20260620 23:40:34.745150 18703 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:40:34.785533 18703 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:40:34.787566 18703 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:40:34.801167 18703 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c71d4c8ad3541049653ecace2c173db: Generated new TSK 0
I20260620 23:40:34.802093 18703 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 23:40:34.874328 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34577
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:40:35.218052 18716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:40:35.218423 18716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:40:35.218600 18716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:40:35.226837 18716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:40:35.227025 18716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:40:35.227198 18716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 23:40:35.237766 18716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34577
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:40:35.239240 18716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:40:35.240633 18716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:40:35.254567 18724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:40:35.254699 18722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:40:35.254653 18721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:40:35.256124 18716 server_base.cc:1034] running on GCE node
I20260620 23:40:35.256853 18716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:40:35.257896 18716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:40:35.259141 18716 hybrid_clock.cc:648] HybridClock initialized: now 1781998835259092 us; error 49 us; skew 500 ppm
I20260620 23:40:35.259527 18716 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:40:35 dist-test-slave-tcgx krb5kdc[18610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:40:35.262828 18716 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 23:40:35.264452 18716 webserver.cc:460] Webserver started at http://127.0.97.1:42697/ using document root <none> and password file <none>
I20260620 23:40:35.265158 18716 fs_manager.cc:359] Metadata directory not provided
I20260620 23:40:35.265262 18716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:40:35.265545 18716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:40:35.268663 18716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/minicluster-data/ts-0/data/instance:
uuid: "3f2e06435db0402aa1be720f8b76e0aa"
format_stamp: "Formatted at 2026-06-20 23:40:35 on dist-test-slave-tcgx"
server_key: "3266b46538325f544073d2ef563d33b3"
server_key_iv: "4d3e2044c81af1aae93dd967f416c2f3"
server_key_version: "encryptionkey@0"
I20260620 23:40:35.269425 18716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/minicluster-data/ts-0/wal/instance:
uuid: "3f2e06435db0402aa1be720f8b76e0aa"
format_stamp: "Formatted at 2026-06-20 23:40:35 on dist-test-slave-tcgx"
server_key: "3266b46538325f544073d2ef563d33b3"
server_key_iv: "4d3e2044c81af1aae93dd967f416c2f3"
server_key_version: "encryptionkey@0"
I20260620 23:40:35.274605 18716 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 23:40:35.278281 18731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:35.279759 18716 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 23:40:35.279963 18716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/minicluster-data/ts-0/wal
uuid: "3f2e06435db0402aa1be720f8b76e0aa"
format_stamp: "Formatted at 2026-06-20 23:40:35 on dist-test-slave-tcgx"
server_key: "3266b46538325f544073d2ef563d33b3"
server_key_iv: "4d3e2044c81af1aae93dd967f416c2f3"
server_key_version: "encryptionkey@0"
I20260620 23:40:35.280180 18716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:40:35.307278 18716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:40:35.311744 18716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:40:35.312060 18716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:40:35.313097 18716 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:40:35.314546 18716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:40:35.314622 18716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:35.314791 18716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:40:35.314850 18716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:35.349481 18716 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33777
I20260620 23:40:35.349579 18843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33777 every 8 connection(s)
I20260620 23:40:35.350862 18716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
I20260620 23:40:35.353130 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 18716
I20260620 23:40:35.353355 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/minicluster-data/ts-0/wal/instance
I20260620 23:40:35.353935 388 external_mini_cluster.cc:1428] Setting key 184c9e4f1218757e6a59f8c57c171999
Jun 20 23:40:35 dist-test-slave-tcgx krb5kdc[18610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:40:35.371680 18846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:35.353563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37585 (local address 127.0.97.62:34577)
0620 23:40:35.354053 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:35.354061 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:35.355261 (+ 1200us) server_negotiation.cc:407] Connection header received
0620 23:40:35.357172 (+ 1911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:35.357178 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:35.357263 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:35.357399 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:40:35.358927 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:35.359715 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:35.361281 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:35.361702 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:35.361839 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:35.365680 (+ 3841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:40:35.365685 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:40:35.365688 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:40:35.365722 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:40:35.368559 (+ 2837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:40:35.369313 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:40:35.369320 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:40:35.369325 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:40:35.369451 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:40:35.369904 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:40:35.369915 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:40:35.369918 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:40:35.370185 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:40:35.370338 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:35.371086 (+ 748us) server_negotiation.cc:299] Negotiation successful
0620 23:40:35.371315 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":222,"threads_started":1}
I20260620 23:40:35.372697 18847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:35.354229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34577 (local address 127.0.97.1:37585)
0620 23:40:35.355091 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:35.355144 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 23:40:35.356899 (+ 1755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:40:35.357637 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:40:35.357646 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:40:35.358187 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:40:35.358749 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:35.358765 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:35.359892 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:35.359896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:35.361131 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:35.361143 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:35.361943 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:35.361949 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:35.362189 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:35.362904 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:40:35.362927 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:40:35.365442 (+ 2515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:40:35.368711 (+ 3269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:40:35.368721 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:40:35.368740 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:40:35.369187 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:40:35.369569 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:40:35.369574 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:40:35.369578 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:40:35.369775 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:40:35.370323 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:40:35.370331 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:40:35.370697 (+ 366us) client_negotiation.cc:769] Sending connection context
0620 23:40:35.370990 (+ 293us) client_negotiation.cc:240] Negotiation successful
0620 23:40:35.371566 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":351,"threads_started":1}
I20260620 23:40:35.374387 18844 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34577
I20260620 23:40:35.374778 18844 heartbeater.cc:464] Registering TS with master...
I20260620 23:40:35.376016 18844 heartbeater.cc:511] Master 127.0.97.62:34577 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 23:40:35.379047 18655 ts_manager.cc:194] Registered new tserver with Master: 3f2e06435db0402aa1be720f8b76e0aa (127.0.97.1:33777)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260620 23:40:35.381222 18655 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:37585
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 23:40:35.418920 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34577
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:40:35.767386 18850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:40:35.767715 18850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:40:35.767885 18850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:40:35.775303 18850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:40:35.775441 18850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:40:35.775633 18850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 23:40:35.785815 18850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34577
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:40:35.787321 18850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:40:35.788686 18850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:40:35.801813 18855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:40:35.801896 18858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:40:35.801924 18856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:40:35.803115 18850 server_base.cc:1034] running on GCE node
I20260620 23:40:35.803740 18850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:40:35.804718 18850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:40:35.805953 18850 hybrid_clock.cc:648] HybridClock initialized: now 1781998835805914 us; error 30 us; skew 500 ppm
I20260620 23:40:35.806288 18850 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:40:35 dist-test-slave-tcgx krb5kdc[18610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:40:35.809441 18850 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 23:40:35.810926 18850 webserver.cc:460] Webserver started at http://127.0.97.2:39027/ using document root <none> and password file <none>
I20260620 23:40:35.811712 18850 fs_manager.cc:359] Metadata directory not provided
I20260620 23:40:35.811858 18850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:40:35.812212 18850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:40:35.815347 18850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/minicluster-data/ts-1/data/instance:
uuid: "96dc503597a8414a877b3c40d6aa45fd"
format_stamp: "Formatted at 2026-06-20 23:40:35 on dist-test-slave-tcgx"
server_key: "64ed9256ed8019436b10ddde355aec93"
server_key_iv: "fd04de524ef452995b88106f6bec0acb"
server_key_version: "encryptionkey@0"
I20260620 23:40:35.816195 18850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/minicluster-data/ts-1/wal/instance:
uuid: "96dc503597a8414a877b3c40d6aa45fd"
format_stamp: "Formatted at 2026-06-20 23:40:35 on dist-test-slave-tcgx"
server_key: "64ed9256ed8019436b10ddde355aec93"
server_key_iv: "fd04de524ef452995b88106f6bec0acb"
server_key_version: "encryptionkey@0"
I20260620 23:40:35.821375 18850 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:40:35.825209 18865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:35.826639 18850 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:40:35.826820 18850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/minicluster-data/ts-1/wal
uuid: "96dc503597a8414a877b3c40d6aa45fd"
format_stamp: "Formatted at 2026-06-20 23:40:35 on dist-test-slave-tcgx"
server_key: "64ed9256ed8019436b10ddde355aec93"
server_key_iv: "fd04de524ef452995b88106f6bec0acb"
server_key_version: "encryptionkey@0"
I20260620 23:40:35.827085 18850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:40:35.843380 18850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:40:35.847956 18850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:40:35.848263 18850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:40:35.849346 18850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:40:35.850744 18850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:40:35.850849 18850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:35.851016 18850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:40:35.851094 18850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:35.885255 18850 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40115
I20260620 23:40:35.885289 18977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40115 every 8 connection(s)
I20260620 23:40:35.886667 18850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
I20260620 23:40:35.897130 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 18850
I20260620 23:40:35.897384 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/minicluster-data/ts-1/wal/instance
I20260620 23:40:35.897831 388 external_mini_cluster.cc:1428] Setting key 4ec7b87cc7aa3369413af7f41f70c6b9
Jun 20 23:40:35 dist-test-slave-tcgx krb5kdc[18610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:40:35.908023 18980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:35.889378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53511 (local address 127.0.97.62:34577)
0620 23:40:35.889830 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:35.889835 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:35.890837 (+ 1002us) server_negotiation.cc:407] Connection header received
0620 23:40:35.892803 (+ 1966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:35.892809 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:35.892901 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:35.893044 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:40:35.894642 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:35.895493 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:35.897414 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:35.897949 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:35.898052 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:35.902122 (+ 4070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:40:35.902128 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:40:35.902131 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:40:35.902165 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:40:35.904763 (+ 2598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:40:35.905551 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:40:35.905557 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:40:35.905561 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:40:35.905700 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:40:35.906128 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:40:35.906133 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:40:35.906136 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:40:35.906420 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:40:35.906577 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:35.907381 (+ 804us) server_negotiation.cc:299] Negotiation successful
0620 23:40:35.907614 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":226,"threads_started":1}
I20260620 23:40:35.909137 18981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:35.889953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34577 (local address 127.0.97.2:53511)
0620 23:40:35.890676 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:35.890714 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 23:40:35.892507 (+ 1793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:40:35.893304 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:40:35.893313 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:40:35.893860 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:40:35.894459 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:35.894477 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:35.895664 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:35.895668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:35.897158 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:35.897176 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:35.898172 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:35.898180 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:35.898497 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:35.899274 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:40:35.899297 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:40:35.901877 (+ 2580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:40:35.904952 (+ 3075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:40:35.904962 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:40:35.904981 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:40:35.905425 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:40:35.905808 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:40:35.905813 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:40:35.905817 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:40:35.906015 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:40:35.906547 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:40:35.906555 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:40:35.906927 (+ 372us) client_negotiation.cc:769] Sending connection context
0620 23:40:35.907302 (+ 375us) client_negotiation.cc:240] Negotiation successful
0620 23:40:35.907889 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":276,"threads_started":1}
I20260620 23:40:35.910815 18978 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34577
I20260620 23:40:35.911164 18978 heartbeater.cc:464] Registering TS with master...
I20260620 23:40:35.911932 18978 heartbeater.cc:511] Master 127.0.97.62:34577 requested a full tablet report, sending...
I20260620 23:40:35.913596 18655 ts_manager.cc:194] Registered new tserver with Master: 96dc503597a8414a877b3c40d6aa45fd (127.0.97.2:40115)
I20260620 23:40:35.914454 18655 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:53511
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 23:40:35.952670 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34577
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:40:36.289083 18984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:40:36.289381 18984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:40:36.289489 18984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:40:36.296867 18984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:40:36.296991 18984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:40:36.297106 18984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 23:40:36.307122 18984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34577
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:40:36.308472 18984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:40:36.309795 18984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:40:36.323401 18989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:40:36.323361 18992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:40:36.323444 18990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:40:36.324451 18984 server_base.cc:1034] running on GCE node
I20260620 23:40:36.325096 18984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:40:36.326109 18984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:40:36.327342 18984 hybrid_clock.cc:648] HybridClock initialized: now 1781998836327272 us; error 60 us; skew 500 ppm
I20260620 23:40:36.327689 18984 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:40:36 dist-test-slave-tcgx krb5kdc[18610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:40:36.330921 18984 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 23:40:36.332647 18984 webserver.cc:460] Webserver started at http://127.0.97.3:37853/ using document root <none> and password file <none>
I20260620 23:40:36.333357 18984 fs_manager.cc:359] Metadata directory not provided
I20260620 23:40:36.333493 18984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:40:36.333814 18984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:40:36.336961 18984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/minicluster-data/ts-2/data/instance:
uuid: "15c60f9bda744775b752db6793a852a5"
format_stamp: "Formatted at 2026-06-20 23:40:36 on dist-test-slave-tcgx"
server_key: "f3f2b9654937552303e00d2e848c4e48"
server_key_iv: "58dd8b513d80a1da7ec09cf29bead638"
server_key_version: "encryptionkey@0"
I20260620 23:40:36.337749 18984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/minicluster-data/ts-2/wal/instance:
uuid: "15c60f9bda744775b752db6793a852a5"
format_stamp: "Formatted at 2026-06-20 23:40:36 on dist-test-slave-tcgx"
server_key: "f3f2b9654937552303e00d2e848c4e48"
server_key_iv: "58dd8b513d80a1da7ec09cf29bead638"
server_key_version: "encryptionkey@0"
I20260620 23:40:36.343212 18984 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260620 23:40:36.346778 18999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:36.348326 18984 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:40:36.348520 18984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/minicluster-data/ts-2/wal
uuid: "15c60f9bda744775b752db6793a852a5"
format_stamp: "Formatted at 2026-06-20 23:40:36 on dist-test-slave-tcgx"
server_key: "f3f2b9654937552303e00d2e848c4e48"
server_key_iv: "58dd8b513d80a1da7ec09cf29bead638"
server_key_version: "encryptionkey@0"
I20260620 23:40:36.348747 18984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:40:36.369254 18984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:40:36.373361 18984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:40:36.373692 18984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:40:36.374701 18984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:40:36.376052 18984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:40:36.376125 18984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:36.376294 18984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:40:36.376344 18984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:36.386310 18844 heartbeater.cc:503] Master 127.0.97.62:34577 was elected leader, sending a full tablet report...
I20260620 23:40:36.411481 18984 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33409
I20260620 23:40:36.411554 19111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33409 every 8 connection(s)
I20260620 23:40:36.412825 18984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
I20260620 23:40:36.418200 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 18984
I20260620 23:40:36.418406 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/minicluster-data/ts-2/wal/instance
I20260620 23:40:36.418988 388 external_mini_cluster.cc:1428] Setting key d9d8934f631d7f0929ca2704aea66462
Jun 20 23:40:36 dist-test-slave-tcgx krb5kdc[18610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:40:36.434725 19114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:36.415536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53487 (local address 127.0.97.62:34577)
0620 23:40:36.415974 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:36.415978 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:36.417158 (+ 1180us) server_negotiation.cc:407] Connection header received
0620 23:40:36.419645 (+ 2487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:36.419654 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:36.419763 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:36.419858 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:40:36.421911 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:36.422726 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:36.424485 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:36.424912 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:36.425009 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:36.428507 (+ 3498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:40:36.428512 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:40:36.428515 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:40:36.428548 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:40:36.431194 (+ 2646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:40:36.432048 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:40:36.432055 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:40:36.432063 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:40:36.432186 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:40:36.432648 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:40:36.432653 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:40:36.432657 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:40:36.433072 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:40:36.433251 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:36.434099 (+ 848us) server_negotiation.cc:299] Negotiation successful
0620 23:40:36.434335 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":201,"threads_started":1}
I20260620 23:40:36.435906 19115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:36.416170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34577 (local address 127.0.97.3:53487)
0620 23:40:36.416982 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:36.417020 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 23:40:36.419294 (+ 2274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:40:36.420148 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:40:36.420158 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:40:36.421007 (+ 849us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:40:36.421675 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:36.421695 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:36.422868 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:36.422872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:36.424353 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:36.424366 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:36.425111 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:36.425115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:36.425330 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:36.426017 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:40:36.426045 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:40:36.428283 (+ 2238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:40:36.431353 (+ 3070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:40:36.431362 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:40:36.431388 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:40:36.431899 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:40:36.432295 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:40:36.432301 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:40:36.432305 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:40:36.432527 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:40:36.433217 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:40:36.433226 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:40:36.433662 (+ 436us) client_negotiation.cc:769] Sending connection context
0620 23:40:36.433982 (+ 320us) client_negotiation.cc:240] Negotiation successful
0620 23:40:36.434535 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":337,"threads_started":1}
I20260620 23:40:36.437297 19112 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34577
I20260620 23:40:36.437650 19112 heartbeater.cc:464] Registering TS with master...
I20260620 23:40:36.438412 19112 heartbeater.cc:511] Master 127.0.97.62:34577 requested a full tablet report, sending...
I20260620 23:40:36.439927 18655 ts_manager.cc:194] Registered new tserver with Master: 15c60f9bda744775b752db6793a852a5 (127.0.97.3:33409)
I20260620 23:40:36.440754 18655 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:53487
I20260620 23:40:36.446906 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:40:36.464344 19114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:36.450989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35786 (local address 127.0.97.62:34577)
0620 23:40:36.451133 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:36.451138 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:36.451556 (+ 418us) server_negotiation.cc:407] Connection header received
0620 23:40:36.451705 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:36.451709 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:36.451827 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:36.451928 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:40:36.453445 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:36.454222 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:36.455629 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:36.456016 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:36.456113 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:36.458675 (+ 2562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:40:36.458680 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:40:36.458684 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:40:36.458727 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:40:36.461490 (+ 2763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:40:36.462314 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:40:36.462318 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:40:36.462321 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:40:36.462445 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:40:36.462849 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:40:36.462853 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:40:36.462855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:40:36.463266 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:40:36.463479 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:36.463803 (+ 324us) server_negotiation.cc:299] Negotiation successful
0620 23:40:36.463985 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 23:40:36.468396 18655 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35786:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:40:36.475492 18655 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:40:36.499347 19128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:36.488772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36390 (local address 127.0.97.2:40115)
0620 23:40:36.489562 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:36.489575 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:36.491046 (+ 1471us) server_negotiation.cc:407] Connection header received
0620 23:40:36.491366 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:36.491389 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:36.491625 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:36.491787 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:40:36.492759 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:36.494285 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:36.496862 (+ 2577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:36.497868 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:36.498004 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:36.498223 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:36.498864 (+ 641us) server_negotiation.cc:299] Negotiation successful
0620 23:40:36.499081 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":436,"threads_started":1}
I20260620 23:40:36.499966 19126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:36.488178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40115 (local address 127.0.0.1:36390)
0620 23:40:36.490900 (+ 2722us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:36.490918 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:40:36.491148 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:40:36.492068 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:40:36.492072 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:40:36.492100 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:40:36.492553 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:36.492566 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:36.494494 (+ 1928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:36.494498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:36.496708 (+ 2210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:36.496723 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:36.498368 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:36.498374 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:36.498575 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:36.498632 (+ 57us) client_negotiation.cc:769] Sending connection context
0620 23:40:36.499583 (+ 951us) client_negotiation.cc:240] Negotiation successful
0620 23:40:36.499724 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2579,"thread_start_us":2410,"threads_started":1}
I20260620 23:40:36.500191 19127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:36.488314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48594 (local address 127.0.97.3:33409)
0620 23:40:36.490059 (+ 1745us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:36.490067 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:36.490100 (+ 33us) server_negotiation.cc:407] Connection header received
0620 23:40:36.490200 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:36.490207 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:36.490479 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:36.490697 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:40:36.491768 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:36.493475 (+ 1707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:36.496712 (+ 3237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:36.498162 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:36.498301 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:36.498576 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:36.499648 (+ 1072us) server_negotiation.cc:299] Negotiation successful
0620 23:40:36.499837 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1596,"thread_start_us":1439,"threads_started":1}
I20260620 23:40:36.500669 19125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:36.488026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33409 (local address 127.0.0.1:48594)
0620 23:40:36.488756 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:36.488798 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 23:40:36.489004 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:40:36.491042 (+ 2038us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:40:36.491049 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:40:36.491081 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:40:36.491553 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:36.491567 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:36.493640 (+ 2073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:36.493645 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:36.496527 (+ 2882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:36.496544 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:36.498722 (+ 2178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:36.498728 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:36.498911 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:36.498968 (+ 57us) client_negotiation.cc:769] Sending connection context
0620 23:40:36.499062 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 23:40:36.499265 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"mutex_wait_us":243,"thread_start_us":304,"threads_started":1}
I20260620 23:40:36.501358 19129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:36.490729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33777 (local address 127.0.0.1:43472)
0620 23:40:36.492817 (+ 2088us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:36.492832 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:40:36.492972 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:40:36.493497 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:40:36.493501 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:40:36.493526 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:40:36.494028 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:36.494041 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:36.496166 (+ 2125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:36.496172 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:36.498314 (+ 2142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:36.498328 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:36.500530 (+ 2202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:36.500536 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:36.500737 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:36.500775 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 23:40:36.500869 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 23:40:36.501024 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1985,"mutex_wait_us":1,"thread_start_us":1835,"threads_started":1}
I20260620 23:40:36.502820 19130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:36.491221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43472 (local address 127.0.97.1:33777)
0620 23:40:36.492326 (+ 1105us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:36.492339 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:36.492942 (+ 603us) server_negotiation.cc:407] Connection header received
0620 23:40:36.493142 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:36.493146 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:36.493316 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:36.493507 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:40:36.494464 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:36.496017 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:36.498502 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:36.500016 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:36.500145 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:36.500383 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:36.501631 (+ 1248us) server_negotiation.cc:299] Negotiation successful
0620 23:40:36.502528 (+ 897us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":917,"thread_start_us":775,"threads_started":1}
I20260620 23:40:36.504205 18913 tablet_service.cc:1467] Processing CreateTablet for tablet 8367eba3a7744249a12a21dc218d7a2c (DEFAULT_TABLE table=test-table [id=ae0993c76963452bb11a83b0e4fec56e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:40:36.504230 19047 tablet_service.cc:1467] Processing CreateTablet for tablet 8367eba3a7744249a12a21dc218d7a2c (DEFAULT_TABLE table=test-table [id=ae0993c76963452bb11a83b0e4fec56e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:40:36.505475 18913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8367eba3a7744249a12a21dc218d7a2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:40:36.505645 19047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8367eba3a7744249a12a21dc218d7a2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:40:36.506810 18779 tablet_service.cc:1467] Processing CreateTablet for tablet 8367eba3a7744249a12a21dc218d7a2c (DEFAULT_TABLE table=test-table [id=ae0993c76963452bb11a83b0e4fec56e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:40:36.509058 18779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8367eba3a7744249a12a21dc218d7a2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:40:36.517237 19131 tablet_bootstrap.cc:492] T 8367eba3a7744249a12a21dc218d7a2c P 96dc503597a8414a877b3c40d6aa45fd: Bootstrap starting.
I20260620 23:40:36.520658 19131 tablet_bootstrap.cc:654] T 8367eba3a7744249a12a21dc218d7a2c P 96dc503597a8414a877b3c40d6aa45fd: Neither blocks nor log segments found. Creating new log.
I20260620 23:40:36.521575 19132 tablet_bootstrap.cc:492] T 8367eba3a7744249a12a21dc218d7a2c P 15c60f9bda744775b752db6793a852a5: Bootstrap starting.
I20260620 23:40:36.522843 19131 log.cc:824] T 8367eba3a7744249a12a21dc218d7a2c P 96dc503597a8414a877b3c40d6aa45fd: Log is configured to *not* fsync() on all Append() calls
I20260620 23:40:36.525413 19132 tablet_bootstrap.cc:654] T 8367eba3a7744249a12a21dc218d7a2c P 15c60f9bda744775b752db6793a852a5: Neither blocks nor log segments found. Creating new log.
I20260620 23:40:36.525563 19134 tablet_bootstrap.cc:492] T 8367eba3a7744249a12a21dc218d7a2c P 3f2e06435db0402aa1be720f8b76e0aa: Bootstrap starting.
I20260620 23:40:36.526261 19131 tablet_bootstrap.cc:492] T 8367eba3a7744249a12a21dc218d7a2c P 96dc503597a8414a877b3c40d6aa45fd: No bootstrap required, opened a new log
I20260620 23:40:36.526556 19131 ts_tablet_manager.cc:1397] T 8367eba3a7744249a12a21dc218d7a2c P 96dc503597a8414a877b3c40d6aa45fd: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260620 23:40:36.526782 19132 log.cc:824] T 8367eba3a7744249a12a21dc218d7a2c P 15c60f9bda744775b752db6793a852a5: Log is configured to *not* fsync() on all Append() calls
I20260620 23:40:36.529120 19134 tablet_bootstrap.cc:654] T 8367eba3a7744249a12a21dc218d7a2c P 3f2e06435db0402aa1be720f8b76e0aa: Neither blocks nor log segments found. Creating new log.
I20260620 23:40:36.529960 19132 tablet_bootstrap.cc:492] T 8367eba3a7744249a12a21dc218d7a2c P 15c60f9bda744775b752db6793a852a5: No bootstrap required, opened a new log
I20260620 23:40:36.530230 19132 ts_tablet_manager.cc:1397] T 8367eba3a7744249a12a21dc218d7a2c P 15c60f9bda744775b752db6793a852a5: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 23:40:36.530217 19134 log.cc:824] T 8367eba3a7744249a12a21dc218d7a2c P 3f2e06435db0402aa1be720f8b76e0aa: Log is configured to *not* fsync() on all Append() calls
I20260620 23:40:36.537369 19134 tablet_bootstrap.cc:492] T 8367eba3a7744249a12a21dc218d7a2c P 3f2e06435db0402aa1be720f8b76e0aa: No bootstrap required, opened a new log
I20260620 23:40:36.537688 19134 ts_tablet_manager.cc:1397] T 8367eba3a7744249a12a21dc218d7a2c P 3f2e06435db0402aa1be720f8b76e0aa: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20260620 23:40:36.537331 19131 raft_consensus.cc:348] T 8367eba3a7744249a12a21dc218d7a2c P 96dc503597a8414a877b3c40d6aa45fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15c60f9bda744775b752db6793a852a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33409 } } peers { permanent_uuid: "96dc503597a8414a877b3c40d6aa45fd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40115 } } peers { permanent_uuid: "3f2e06435db0402aa1be720f8b76e0aa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33777 } }
I20260620 23:40:36.537847 19131 raft_consensus.cc:374] T 8367eba3a7744249a12a21dc218d7a2c P 96dc503597a8414a877b3c40d6aa45fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:40:36.537954 19131 raft_consensus.cc:729] T 8367eba3a7744249a12a21dc218d7a2c P 96dc503597a8414a877b3c40d6aa45fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96dc503597a8414a877b3c40d6aa45fd, State: Initialized, Role: FOLLOWER
I20260620 23:40:36.538637 19131 consensus_queue.cc:260] T 8367eba3a7744249a12a21dc218d7a2c P 96dc503597a8414a877b3c40d6aa45fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15c60f9bda744775b752db6793a852a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33409 } } peers { permanent_uuid: "96dc503597a8414a877b3c40d6aa45fd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40115 } } peers { permanent_uuid: "3f2e06435db0402aa1be720f8b76e0aa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33777 } }
I20260620 23:40:36.539911 18978 heartbeater.cc:503] Master 127.0.97.62:34577 was elected leader, sending a full tablet report...
I20260620 23:40:36.540577 19131 ts_tablet_manager.cc:1428] T 8367eba3a7744249a12a21dc218d7a2c P 96dc503597a8414a877b3c40d6aa45fd: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260620 23:40:36.541363 19132 raft_consensus.cc:348] T 8367eba3a7744249a12a21dc218d7a2c P 15c60f9bda744775b752db6793a852a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15c60f9bda744775b752db6793a852a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33409 } } peers { permanent_uuid: "96dc503597a8414a877b3c40d6aa45fd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40115 } } peers { permanent_uuid: "3f2e06435db0402aa1be720f8b76e0aa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33777 } }
I20260620 23:40:36.541803 19132 raft_consensus.cc:374] T 8367eba3a7744249a12a21dc218d7a2c P 15c60f9bda744775b752db6793a852a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:40:36.541939 19132 raft_consensus.cc:729] T 8367eba3a7744249a12a21dc218d7a2c P 15c60f9bda744775b752db6793a852a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15c60f9bda744775b752db6793a852a5, State: Initialized, Role: FOLLOWER
I20260620 23:40:36.542591 19132 consensus_queue.cc:260] T 8367eba3a7744249a12a21dc218d7a2c P 15c60f9bda744775b752db6793a852a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15c60f9bda744775b752db6793a852a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33409 } } peers { permanent_uuid: "96dc503597a8414a877b3c40d6aa45fd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40115 } } peers { permanent_uuid: "3f2e06435db0402aa1be720f8b76e0aa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33777 } }
I20260620 23:40:36.544720 19112 heartbeater.cc:503] Master 127.0.97.62:34577 was elected leader, sending a full tablet report...
I20260620 23:40:36.545490 19132 ts_tablet_manager.cc:1428] T 8367eba3a7744249a12a21dc218d7a2c P 15c60f9bda744775b752db6793a852a5: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260620 23:40:36.548763 19134 raft_consensus.cc:348] T 8367eba3a7744249a12a21dc218d7a2c P 3f2e06435db0402aa1be720f8b76e0aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15c60f9bda744775b752db6793a852a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33409 } } peers { permanent_uuid: "96dc503597a8414a877b3c40d6aa45fd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40115 } } peers { permanent_uuid: "3f2e06435db0402aa1be720f8b76e0aa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33777 } }
I20260620 23:40:36.549222 19134 raft_consensus.cc:374] T 8367eba3a7744249a12a21dc218d7a2c P 3f2e06435db0402aa1be720f8b76e0aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:40:36.549337 19134 raft_consensus.cc:729] T 8367eba3a7744249a12a21dc218d7a2c P 3f2e06435db0402aa1be720f8b76e0aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f2e06435db0402aa1be720f8b76e0aa, State: Initialized, Role: FOLLOWER
I20260620 23:40:36.549952 19134 consensus_queue.cc:260] T 8367eba3a7744249a12a21dc218d7a2c P 3f2e06435db0402aa1be720f8b76e0aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15c60f9bda744775b752db6793a852a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33409 } } peers { permanent_uuid: "96dc503597a8414a877b3c40d6aa45fd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40115 } } peers { permanent_uuid: "3f2e06435db0402aa1be720f8b76e0aa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33777 } }
I20260620 23:40:36.551478 19134 ts_tablet_manager.cc:1428] T 8367eba3a7744249a12a21dc218d7a2c P 3f2e06435db0402aa1be720f8b76e0aa: Time spent starting tablet: real 0.014s user 0.013s sys 0.001s
I20260620 23:40:36.565392 19139 raft_consensus.cc:482] T 8367eba3a7744249a12a21dc218d7a2c P 3f2e06435db0402aa1be720f8b76e0aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:40:36.565682 19139 raft_consensus.cc:504] T 8367eba3a7744249a12a21dc218d7a2c P 3f2e06435db0402aa1be720f8b76e0aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15c60f9bda744775b752db6793a852a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33409 } } peers { permanent_uuid: "96dc503597a8414a877b3c40d6aa45fd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40115 } } peers { permanent_uuid: "3f2e06435db0402aa1be720f8b76e0aa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33777 } }
I20260620 23:40:36.567482 19139 leader_election.cc:288] T 8367eba3a7744249a12a21dc218d7a2c P 3f2e06435db0402aa1be720f8b76e0aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15c60f9bda744775b752db6793a852a5 (127.0.97.3:33409), 96dc503597a8414a877b3c40d6aa45fd (127.0.97.2:40115)
I20260620 23:40:36.574873 19128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:36.567923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43587 (local address 127.0.97.2:40115)
0620 23:40:36.568156 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:36.568166 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:36.569179 (+ 1013us) server_negotiation.cc:407] Connection header received
0620 23:40:36.569363 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:36.569369 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:36.569438 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:36.569535 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:40:36.570146 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:36.570837 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:36.573071 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:36.573838 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:36.573923 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:36.573993 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:36.574478 (+ 485us) server_negotiation.cc:299] Negotiation successful
0620 23:40:36.574609 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 23:40:36.574888 19127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:36.567941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60877 (local address 127.0.97.3:33409)
0620 23:40:36.568156 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:36.568165 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:36.569035 (+ 870us) server_negotiation.cc:407] Connection header received
0620 23:40:36.569269 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:36.569273 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:36.569332 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:36.569451 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:40:36.570022 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:36.570770 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:36.573098 (+ 2328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:36.573874 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:36.574033 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:36.574141 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:36.574507 (+ 366us) server_negotiation.cc:299] Negotiation successful
0620 23:40:36.574633 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 23:40:36.575330 19141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:36.568021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40115 (local address 127.0.97.1:43587)
0620 23:40:36.569069 (+ 1048us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:36.569083 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:40:36.569228 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:40:36.569547 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:40:36.569555 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:40:36.569590 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:40:36.570002 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:36.570011 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:36.571009 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:36.571014 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:36.572868 (+ 1854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:36.572882 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:36.574157 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:36.574161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:36.574320 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:36.574343 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 23:40:36.574453 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 23:40:36.574582 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":947,"mutex_wait_us":52,"thread_start_us":696,"threads_started":1}
I20260620 23:40:36.575371 19140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:36.567863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33409 (local address 127.0.97.1:60877)
0620 23:40:36.568906 (+ 1043us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:36.568934 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 23:40:36.569120 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:40:36.569464 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:40:36.569469 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:40:36.569501 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:40:36.569858 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:36.569869 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:36.570900 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:36.570907 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:36.572867 (+ 1960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:36.572885 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:36.574097 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:36.574107 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:36.574337 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:36.574361 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 23:40:36.574469 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 23:40:36.574596 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":887,"mutex_wait_us":102,"thread_start_us":661,"threads_started":1}
I20260620 23:40:36.576332 18933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8367eba3a7744249a12a21dc218d7a2c" candidate_uuid: "3f2e06435db0402aa1be720f8b76e0aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96dc503597a8414a877b3c40d6aa45fd" is_pre_election: true
I20260620 23:40:36.576331 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8367eba3a7744249a12a21dc218d7a2c" candidate_uuid: "3f2e06435db0402aa1be720f8b76e0aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15c60f9bda744775b752db6793a852a5" is_pre_election: true
I20260620 23:40:36.576828 18933 raft_consensus.cc:2454] T 8367eba3a7744249a12a21dc218d7a2c P 96dc503597a8414a877b3c40d6aa45fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f2e06435db0402aa1be720f8b76e0aa in term 0.
I20260620 23:40:36.576829 19067 raft_consensus.cc:2454] T 8367eba3a7744249a12a21dc218d7a2c P 15c60f9bda744775b752db6793a852a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f2e06435db0402aa1be720f8b76e0aa in term 0.
I20260620 23:40:36.577561 18732 leader_election.cc:302] T 8367eba3a7744249a12a21dc218d7a2c P 3f2e06435db0402aa1be720f8b76e0aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 15c60f9bda744775b752db6793a852a5, 3f2e06435db0402aa1be720f8b76e0aa, 96dc503597a8414a877b3c40d6aa45fd; no voters:
I20260620 23:40:36.577979 19139 raft_consensus.cc:2781] T 8367eba3a7744249a12a21dc218d7a2c P 3f2e06435db0402aa1be720f8b76e0aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:40:36.578126 19139 raft_consensus.cc:482] T 8367eba3a7744249a12a21dc218d7a2c P 3f2e06435db0402aa1be720f8b76e0aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:40:36.578227 19139 raft_consensus.cc:3037] T 8367eba3a7744249a12a21dc218d7a2c P 3f2e06435db0402aa1be720f8b76e0aa [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:40:36.580641 19139 raft_consensus.cc:504] T 8367eba3a7744249a12a21dc218d7a2c P 3f2e06435db0402aa1be720f8b76e0aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15c60f9bda744775b752db6793a852a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33409 } } peers { permanent_uuid: "96dc503597a8414a877b3c40d6aa45fd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40115 } } peers { permanent_uuid: "3f2e06435db0402aa1be720f8b76e0aa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33777 } }
I20260620 23:40:36.581296 19139 leader_election.cc:288] T 8367eba3a7744249a12a21dc218d7a2c P 3f2e06435db0402aa1be720f8b76e0aa [CANDIDATE]: Term 1 election: Requested vote from peers 15c60f9bda744775b752db6793a852a5 (127.0.97.3:33409), 96dc503597a8414a877b3c40d6aa45fd (127.0.97.2:40115)
I20260620 23:40:36.581817 18933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8367eba3a7744249a12a21dc218d7a2c" candidate_uuid: "3f2e06435db0402aa1be720f8b76e0aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96dc503597a8414a877b3c40d6aa45fd"
I20260620 23:40:36.581817 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8367eba3a7744249a12a21dc218d7a2c" candidate_uuid: "3f2e06435db0402aa1be720f8b76e0aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15c60f9bda744775b752db6793a852a5"
I20260620 23:40:36.582052 19067 raft_consensus.cc:3037] T 8367eba3a7744249a12a21dc218d7a2c P 15c60f9bda744775b752db6793a852a5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:40:36.582052 18933 raft_consensus.cc:3037] T 8367eba3a7744249a12a21dc218d7a2c P 96dc503597a8414a877b3c40d6aa45fd [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:40:36.585461 19067 raft_consensus.cc:2454] T 8367eba3a7744249a12a21dc218d7a2c P 15c60f9bda744775b752db6793a852a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f2e06435db0402aa1be720f8b76e0aa in term 1.
I20260620 23:40:36.585466 18933 raft_consensus.cc:2454] T 8367eba3a7744249a12a21dc218d7a2c P 96dc503597a8414a877b3c40d6aa45fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f2e06435db0402aa1be720f8b76e0aa in term 1.
I20260620 23:40:36.585974 18733 leader_election.cc:302] T 8367eba3a7744249a12a21dc218d7a2c P 3f2e06435db0402aa1be720f8b76e0aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f2e06435db0402aa1be720f8b76e0aa, 96dc503597a8414a877b3c40d6aa45fd; no voters:
I20260620 23:40:36.586301 19139 raft_consensus.cc:2781] T 8367eba3a7744249a12a21dc218d7a2c P 3f2e06435db0402aa1be720f8b76e0aa [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:40:36.586761 19139 raft_consensus.cc:686] T 8367eba3a7744249a12a21dc218d7a2c P 3f2e06435db0402aa1be720f8b76e0aa [term 1 LEADER]: Becoming Leader. State: Replica: 3f2e06435db0402aa1be720f8b76e0aa, State: Running, Role: LEADER
I20260620 23:40:36.587303 19139 consensus_queue.cc:237] T 8367eba3a7744249a12a21dc218d7a2c P 3f2e06435db0402aa1be720f8b76e0aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15c60f9bda744775b752db6793a852a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33409 } } peers { permanent_uuid: "96dc503597a8414a877b3c40d6aa45fd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40115 } } peers { permanent_uuid: "3f2e06435db0402aa1be720f8b76e0aa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33777 } }
I20260620 23:40:36.592155 18654 catalog_manager.cc:5495] T 8367eba3a7744249a12a21dc218d7a2c P 3f2e06435db0402aa1be720f8b76e0aa reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f2e06435db0402aa1be720f8b76e0aa (127.0.97.1). New cstate: current_term: 1 leader_uuid: "3f2e06435db0402aa1be720f8b76e0aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15c60f9bda744775b752db6793a852a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96dc503597a8414a877b3c40d6aa45fd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f2e06435db0402aa1be720f8b76e0aa" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33777 } health_report { overall_health: HEALTHY } } }
W20260620 23:40:36.603328 18845 tablet.cc:2369] T 8367eba3a7744249a12a21dc218d7a2c P 3f2e06435db0402aa1be720f8b76e0aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:40:36.630718 19130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:36.625015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43480 (local address 127.0.97.1:33777)
0620 23:40:36.625244 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:36.625250 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:36.625361 (+ 111us) server_negotiation.cc:407] Connection header received
0620 23:40:36.625611 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:36.625615 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:36.625693 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:36.625843 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:40:36.626328 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:36.627080 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:36.628468 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:36.628866 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:36.629006 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:36.629603 (+ 597us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:40:36.630128 (+ 525us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:40:36.630320 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:36.630436 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 23:40:36.630485 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
W20260620 23:40:36.638692 18979 tablet.cc:2369] T 8367eba3a7744249a12a21dc218d7a2c P 96dc503597a8414a877b3c40d6aa45fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:40:36.639426 19067 raft_consensus.cc:1261] T 8367eba3a7744249a12a21dc218d7a2c P 15c60f9bda744775b752db6793a852a5 [term 1 FOLLOWER]: Refusing update from remote peer 3f2e06435db0402aa1be720f8b76e0aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:40:36.639503 18933 raft_consensus.cc:1261] T 8367eba3a7744249a12a21dc218d7a2c P 96dc503597a8414a877b3c40d6aa45fd [term 1 FOLLOWER]: Refusing update from remote peer 3f2e06435db0402aa1be720f8b76e0aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:40:36.640506 19139 consensus_queue.cc:1035] T 8367eba3a7744249a12a21dc218d7a2c P 3f2e06435db0402aa1be720f8b76e0aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "96dc503597a8414a877b3c40d6aa45fd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:40:36.640802 19142 consensus_queue.cc:1035] T 8367eba3a7744249a12a21dc218d7a2c P 3f2e06435db0402aa1be720f8b76e0aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "15c60f9bda744775b752db6793a852a5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 23:40:36.664455 19113 tablet.cc:2369] T 8367eba3a7744249a12a21dc218d7a2c P 15c60f9bda744775b752db6793a852a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:40:36.665333 19146 mvcc.cc:205] Tried to move back new op lower bound from 7299067234864136192 to 7299067234665455616. Current Snapshot: MvccSnapshot[applied={T|T < 7299067234864136192}]
I20260620 23:40:36.687541 18655 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35786:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:40:36.687922 18655 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35786:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:40:36.688804 19147 mvcc.cc:205] Tried to move back new op lower bound from 7299067234864136192 to 7299067234665455616. Current Snapshot: MvccSnapshot[applied={T|T < 7299067234864136192}]
I20260620 23:40:36.692345 18655 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3c71d4c8ad3541049653ecace2c173db: Sending DeleteTablet for 3 replicas of tablet 8367eba3a7744249a12a21dc218d7a2c
I20260620 23:40:36.693553 19047 tablet_service.cc:1514] Processing DeleteTablet for tablet 8367eba3a7744249a12a21dc218d7a2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:40:36 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48594
I20260620 23:40:36.697551 19155 tablet_replica.cc:330] T 8367eba3a7744249a12a21dc218d7a2c P 15c60f9bda744775b752db6793a852a5: stopping tablet replica
I20260620 23:40:36.697768 18913 tablet_service.cc:1514] Processing DeleteTablet for tablet 8367eba3a7744249a12a21dc218d7a2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:40:36 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36390
I20260620 23:40:36.698032 19155 raft_consensus.cc:2229] T 8367eba3a7744249a12a21dc218d7a2c P 15c60f9bda744775b752db6793a852a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:40:36.698694 19155 raft_consensus.cc:2258] T 8367eba3a7744249a12a21dc218d7a2c P 15c60f9bda744775b752db6793a852a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:40:36.700970 19155 ts_tablet_manager.cc:1905] T 8367eba3a7744249a12a21dc218d7a2c P 15c60f9bda744775b752db6793a852a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:40:36.701406 18779 tablet_service.cc:1514] Processing DeleteTablet for tablet 8367eba3a7744249a12a21dc218d7a2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:40:36 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43472
I20260620 23:40:36.709908 19155 ts_tablet_manager.cc:1918] T 8367eba3a7744249a12a21dc218d7a2c P 15c60f9bda744775b752db6793a852a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:40:36.710093 19155 log.cc:1196] T 8367eba3a7744249a12a21dc218d7a2c P 15c60f9bda744775b752db6793a852a5: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/minicluster-data/ts-2/wal/wals/8367eba3a7744249a12a21dc218d7a2c
I20260620 23:40:36.710649 19155 ts_tablet_manager.cc:1939] T 8367eba3a7744249a12a21dc218d7a2c P 15c60f9bda744775b752db6793a852a5: Deleting consensus metadata
I20260620 23:40:36.712503 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 18716
I20260620 23:40:36.712765 18639 catalog_manager.cc:4841] TS 15c60f9bda744775b752db6793a852a5 (127.0.97.3:33409): tablet 8367eba3a7744249a12a21dc218d7a2c (table test-table [id=ae0993c76963452bb11a83b0e4fec56e]) successfully deleted
I20260620 23:40:36.717725 19157 tablet_replica.cc:330] T 8367eba3a7744249a12a21dc218d7a2c P 3f2e06435db0402aa1be720f8b76e0aa: stopping tablet replica
I20260620 23:40:36.718345 19157 raft_consensus.cc:2229] T 8367eba3a7744249a12a21dc218d7a2c P 3f2e06435db0402aa1be720f8b76e0aa [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:40:36.719228 19157 raft_consensus.cc:2258] T 8367eba3a7744249a12a21dc218d7a2c P 3f2e06435db0402aa1be720f8b76e0aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:40:36.720319 19156 tablet_replica.cc:330] T 8367eba3a7744249a12a21dc218d7a2c P 96dc503597a8414a877b3c40d6aa45fd: stopping tablet replica
I20260620 23:40:36.720759 19156 raft_consensus.cc:2229] T 8367eba3a7744249a12a21dc218d7a2c P 96dc503597a8414a877b3c40d6aa45fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:40:36.721273 19156 raft_consensus.cc:2258] T 8367eba3a7744249a12a21dc218d7a2c P 96dc503597a8414a877b3c40d6aa45fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:40:36.722293 19157 ts_tablet_manager.cc:1905] T 8367eba3a7744249a12a21dc218d7a2c P 3f2e06435db0402aa1be720f8b76e0aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:40:36.725912 19156 ts_tablet_manager.cc:1905] T 8367eba3a7744249a12a21dc218d7a2c P 96dc503597a8414a877b3c40d6aa45fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:40:36.728159 19157 ts_tablet_manager.cc:1918] T 8367eba3a7744249a12a21dc218d7a2c P 3f2e06435db0402aa1be720f8b76e0aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:40:36.728349 19157 log.cc:1196] T 8367eba3a7744249a12a21dc218d7a2c P 3f2e06435db0402aa1be720f8b76e0aa: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/minicluster-data/ts-0/wal/wals/8367eba3a7744249a12a21dc218d7a2c
I20260620 23:40:36.728986 19157 ts_tablet_manager.cc:1939] T 8367eba3a7744249a12a21dc218d7a2c P 3f2e06435db0402aa1be720f8b76e0aa: Deleting consensus metadata
I20260620 23:40:36.731299 18639 catalog_manager.cc:4841] TS 3f2e06435db0402aa1be720f8b76e0aa (127.0.97.1:33777): tablet 8367eba3a7744249a12a21dc218d7a2c (table test-table [id=ae0993c76963452bb11a83b0e4fec56e]) successfully deleted
I20260620 23:40:36.737043 19156 ts_tablet_manager.cc:1918] T 8367eba3a7744249a12a21dc218d7a2c P 96dc503597a8414a877b3c40d6aa45fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:40:36.737227 19156 log.cc:1196] T 8367eba3a7744249a12a21dc218d7a2c P 96dc503597a8414a877b3c40d6aa45fd: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998525501163-388-0/minicluster-data/ts-1/wal/wals/8367eba3a7744249a12a21dc218d7a2c
I20260620 23:40:36.737809 19156 ts_tablet_manager.cc:1939] T 8367eba3a7744249a12a21dc218d7a2c P 96dc503597a8414a877b3c40d6aa45fd: Deleting consensus metadata
I20260620 23:40:36.739933 18640 catalog_manager.cc:4841] TS 96dc503597a8414a877b3c40d6aa45fd (127.0.97.2:40115): tablet 8367eba3a7744249a12a21dc218d7a2c (table test-table [id=ae0993c76963452bb11a83b0e4fec56e]) successfully deleted
Jun 20 23:40:36 dist-test-slave-tcgx krb5kdc[18610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 23:40:36.752080 19130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:36.714893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43486 (local address 127.0.97.1:33777)
0620 23:40:36.715194 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:36.715203 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:36.722098 (+ 6895us) server_negotiation.cc:407] Connection header received
0620 23:40:36.723067 (+ 969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:36.723074 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:36.723168 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:36.723252 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:40:36.730255 (+ 7003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:36.731582 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:36.733772 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:36.734668 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:36.734819 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:36.741846 (+ 7027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:40:36.741866 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:40:36.741889 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:40:36.741964 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:40:36.746965 (+ 5001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:40:36.748133 (+ 1168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:40:36.748147 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:40:36.748153 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:40:36.748385 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:40:36.748922 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:40:36.748930 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:40:36.748935 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:40:36.749764 (+ 829us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:40:36.750280 (+ 516us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:36.751168 (+ 888us) server_negotiation.cc:299] Negotiation successful
0620 23:40:36.751517 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260620 23:40:36.754053 18839 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:40:36.863698 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 18716
I20260620 23:40:36.892306 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 18850
Jun 20 23:40:36 dist-test-slave-tcgx krb5kdc[18610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 23:40:36.912271 19128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:36.893086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36402 (local address 127.0.97.2:40115)
0620 23:40:36.893312 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:36.893321 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:36.893347 (+ 26us) server_negotiation.cc:407] Connection header received
0620 23:40:36.893517 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:36.893526 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:36.893645 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:36.893784 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:40:36.895437 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:36.896922 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:36.898853 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:36.899748 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:36.899898 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:36.905790 (+ 5892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:40:36.905806 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:40:36.905825 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:40:36.905868 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:40:36.909050 (+ 3182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:40:36.910097 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:40:36.910104 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:40:36.910106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:40:36.910228 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:40:36.910708 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:40:36.910712 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:40:36.910714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:40:36.911169 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:40:36.911405 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:36.911836 (+ 431us) server_negotiation.cc:299] Negotiation successful
0620 23:40:36.912016 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 23:40:36.912992 18973 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:40:37.006695 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 18850
I20260620 23:40:37.023888 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 18984
Jun 20 23:40:37 dist-test-slave-tcgx krb5kdc[18610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 23:40:37.038986 19127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:37.024669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48600 (local address 127.0.97.3:33409)
0620 23:40:37.024880 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:37.024885 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:37.024904 (+ 19us) server_negotiation.cc:407] Connection header received
0620 23:40:37.025005 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:37.025009 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:37.025075 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:37.025201 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:40:37.026628 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:37.027417 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:37.028701 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:37.029083 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:37.029239 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:37.033845 (+ 4606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:40:37.033861 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:40:37.033876 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:40:37.033911 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:40:37.036565 (+ 2654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:40:37.037223 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:40:37.037230 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:40:37.037232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:40:37.037357 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:40:37.037672 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:40:37.037675 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:40:37.037677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:40:37.038014 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:40:37.038223 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:37.038514 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 23:40:37.038677 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260620 23:40:37.039672 19107 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:40:37.129765 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 18984
I20260620 23:40:37.146430 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 18623
I20260620 23:40:37.147614 18685 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:40:37.260465 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 18623
2026-06-20T23:40:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4434 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T23:40:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:40:37Z Disabled control of system clock
I20260620 23:40:37.338519 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998525501163-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998525501163-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998525501163-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39527
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:35429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39527
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 23:40:37.682179 19166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:40:37.682516 19166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:40:37.682602 19166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:40:37.691509 19166 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:40:37.691627 19166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:40:37.691665 19166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:40:37.691723 19166 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:40:37.691768 19166 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:40:37.701334 19166 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998525501163-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998525501163-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39527
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998525501163-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:40:37.702667 19166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:40:37.704510 19166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:40:37.717098 19172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:40:37.717190 19174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:40:37.717167 19171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:40:37.717919 19166 server_base.cc:1034] running on GCE node
I20260620 23:40:37.718792 19166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:40:37.720440 19166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:40:37.721670 19166 hybrid_clock.cc:648] HybridClock initialized: now 1781998837721614 us; error 52 us; skew 500 ppm
I20260620 23:40:37.722054 19166 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:40:37.724556 19166 webserver.cc:460] Webserver started at http://127.0.97.62:43473/ using document root <none> and password file <none>
I20260620 23:40:37.725302 19166 fs_manager.cc:359] Metadata directory not provided
I20260620 23:40:37.725437 19166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:40:37.725807 19166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:40:37.729012 19166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998525501163-388-0/minicluster-data/master-0/data/instance:
uuid: "f9a130e888ee412fa68f21558a38fab6"
format_stamp: "Formatted at 2026-06-20 23:40:37 on dist-test-slave-tcgx"
server_key: "fae653f37c2fcc53a506a036efab41b2"
server_key_iv: "183fc4bbcee6628ecbe7ffc7836d5b94"
server_key_version: "encryptionkey@0"
I20260620 23:40:37.729867 19166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998525501163-388-0/minicluster-data/master-0/wal/instance:
uuid: "f9a130e888ee412fa68f21558a38fab6"
format_stamp: "Formatted at 2026-06-20 23:40:37 on dist-test-slave-tcgx"
server_key: "fae653f37c2fcc53a506a036efab41b2"
server_key_iv: "183fc4bbcee6628ecbe7ffc7836d5b94"
server_key_version: "encryptionkey@0"
I20260620 23:40:37.735558 19166 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260620 23:40:37.739323 19180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:37.740803 19166 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 23:40:37.741014 19166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998525501163-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998525501163-388-0/minicluster-data/master-0/wal
uuid: "f9a130e888ee412fa68f21558a38fab6"
format_stamp: "Formatted at 2026-06-20 23:40:37 on dist-test-slave-tcgx"
server_key: "fae653f37c2fcc53a506a036efab41b2"
server_key_iv: "183fc4bbcee6628ecbe7ffc7836d5b94"
server_key_version: "encryptionkey@0"
I20260620 23:40:37.741261 19166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998525501163-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998525501163-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998525501163-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:40:37.771072 19166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:40:37.772055 19166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:40:37.772372 19166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:40:37.791518 19166 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39527
I20260620 23:40:37.791536 19231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39527 every 8 connection(s)
I20260620 23:40:37.793013 19166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:40:37.796962 19232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:40:37.803054 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 19166
I20260620 23:40:37.803248 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998525501163-388-0/minicluster-data/master-0/wal/instance
I20260620 23:40:37.803680 388 external_mini_cluster.cc:1428] Setting key d0cc79d95605e6798f2c8a1cc5816b98
I20260620 23:40:37.809971 19232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9a130e888ee412fa68f21558a38fab6: Bootstrap starting.
I20260620 23:40:37.813982 19232 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9a130e888ee412fa68f21558a38fab6: Neither blocks nor log segments found. Creating new log.
I20260620 23:40:37.815364 19232 log.cc:824] T 00000000000000000000000000000000 P f9a130e888ee412fa68f21558a38fab6: Log is configured to *not* fsync() on all Append() calls
I20260620 23:40:37.818706 19235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:37.805828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44052 (local address 127.0.97.62:39527)
0620 23:40:37.806977 (+ 1149us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:37.807000 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:37.807061 (+ 61us) server_negotiation.cc:407] Connection header received
0620 23:40:37.808928 (+ 1867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:37.808954 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:37.809457 (+ 503us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:37.809830 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:40:37.810488 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:37.811876 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:37.813789 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:37.814576 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:37.814707 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:37.816132 (+ 1425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:40:37.816147 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:40:37.816171 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:40:37.816966 (+ 795us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:40:37.817046 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:40:37.817071 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:40:37.817157 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:37.817404 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 23:40:37.817754 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":658,"thread_start_us":393,"threads_started":1}
I20260620 23:40:37.818799 19232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9a130e888ee412fa68f21558a38fab6: No bootstrap required, opened a new log
I20260620 23:40:37.827757 19232 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9a130e888ee412fa68f21558a38fab6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9a130e888ee412fa68f21558a38fab6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39527 } }
I20260620 23:40:37.828198 19232 raft_consensus.cc:374] T 00000000000000000000000000000000 P f9a130e888ee412fa68f21558a38fab6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:40:37.828328 19232 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9a130e888ee412fa68f21558a38fab6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9a130e888ee412fa68f21558a38fab6, State: Initialized, Role: FOLLOWER
I20260620 23:40:37.828924 19232 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9a130e888ee412fa68f21558a38fab6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9a130e888ee412fa68f21558a38fab6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39527 } }
I20260620 23:40:37.829191 19232 raft_consensus.cc:388] T 00000000000000000000000000000000 P f9a130e888ee412fa68f21558a38fab6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:40:37.829334 19232 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9a130e888ee412fa68f21558a38fab6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:40:37.829517 19232 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9a130e888ee412fa68f21558a38fab6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:40:37.832171 19232 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9a130e888ee412fa68f21558a38fab6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9a130e888ee412fa68f21558a38fab6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39527 } }
I20260620 23:40:37.832724 19232 leader_election.cc:302] T 00000000000000000000000000000000 P f9a130e888ee412fa68f21558a38fab6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f9a130e888ee412fa68f21558a38fab6; no voters:
I20260620 23:40:37.833329 19232 leader_election.cc:288] T 00000000000000000000000000000000 P f9a130e888ee412fa68f21558a38fab6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:40:37.833505 19237 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9a130e888ee412fa68f21558a38fab6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:40:37.834211 19237 raft_consensus.cc:686] T 00000000000000000000000000000000 P f9a130e888ee412fa68f21558a38fab6 [term 1 LEADER]: Becoming Leader. State: Replica: f9a130e888ee412fa68f21558a38fab6, State: Running, Role: LEADER
I20260620 23:40:37.834776 19237 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9a130e888ee412fa68f21558a38fab6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9a130e888ee412fa68f21558a38fab6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39527 } }
I20260620 23:40:37.835561 19232 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9a130e888ee412fa68f21558a38fab6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:40:37.838023 19238 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9a130e888ee412fa68f21558a38fab6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f9a130e888ee412fa68f21558a38fab6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9a130e888ee412fa68f21558a38fab6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39527 } } }
I20260620 23:40:37.838562 19238 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9a130e888ee412fa68f21558a38fab6 [sys.catalog]: This master's current role is: LEADER
I20260620 23:40:37.838065 19239 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9a130e888ee412fa68f21558a38fab6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9a130e888ee412fa68f21558a38fab6. Latest consensus state: current_term: 1 leader_uuid: "f9a130e888ee412fa68f21558a38fab6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9a130e888ee412fa68f21558a38fab6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39527 } } }
I20260620 23:40:37.839124 19239 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9a130e888ee412fa68f21558a38fab6 [sys.catalog]: This master's current role is: LEADER
I20260620 23:40:37.842137 19245 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:40:37.847114 19245 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:40:37.855527 19245 catalog_manager.cc:1347] Generated new cluster ID: 7d6bbfbb168c4a82bec8e11a22806dbb
I20260620 23:40:37.855670 19245 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:40:37.864710 19245 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:40:37.866053 19245 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:40:37.873499 19245 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f9a130e888ee412fa68f21558a38fab6: Generated new TSK 0
I20260620 23:40:37.874655 19245 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:40:37.889199 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998525501163-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998525501163-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998525501163-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39527
--builtin_ntp_servers=127.0.97.20:35429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 23:40:38.233925 19256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:40:38.234241 19256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:40:38.234419 19256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:40:38.242357 19256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:40:38.242483 19256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:40:38.242656 19256 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 23:40:38.252346 19256 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998525501163-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998525501163-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39527
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998525501163-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:40:38.253746 19256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:40:38.255577 19256 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:40:38.268894 19264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:40:38.268970 19262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:40:38.268862 19261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:40:38.275302 19256 server_base.cc:1034] running on GCE node
I20260620 23:40:38.276130 19256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:40:38.277165 19256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:40:38.278374 19256 hybrid_clock.cc:648] HybridClock initialized: now 1781998838278326 us; error 40 us; skew 500 ppm
I20260620 23:40:38.278723 19256 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:40:38.281220 19256 webserver.cc:460] Webserver started at http://127.0.97.1:42763/ using document root <none> and password file <none>
I20260620 23:40:38.281950 19256 fs_manager.cc:359] Metadata directory not provided
I20260620 23:40:38.282071 19256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:40:38.282423 19256 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:40:38.285554 19256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998525501163-388-0/minicluster-data/ts-0/data/instance:
uuid: "c84eda7f94864d5d86f4fba4a485f01b"
format_stamp: "Formatted at 2026-06-20 23:40:38 on dist-test-slave-tcgx"
server_key: "0d6229970bb03f7ff78a6f1b1f857990"
server_key_iv: "73ad8e1040014925d0e1defae4b3a146"
server_key_version: "encryptionkey@0"
I20260620 23:40:38.286464 19256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998525501163-388-0/minicluster-data/ts-0/wal/instance:
uuid: "c84eda7f94864d5d86f4fba4a485f01b"
format_stamp: "Formatted at 2026-06-20 23:40:38 on dist-test-slave-tcgx"
server_key: "0d6229970bb03f7ff78a6f1b1f857990"
server_key_iv: "73ad8e1040014925d0e1defae4b3a146"
server_key_version: "encryptionkey@0"
I20260620 23:40:38.291889 19256 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 23:40:38.295511 19270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:38.296962 19256 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 23:40:38.297142 19256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998525501163-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998525501163-388-0/minicluster-data/ts-0/wal
uuid: "c84eda7f94864d5d86f4fba4a485f01b"
format_stamp: "Formatted at 2026-06-20 23:40:38 on dist-test-slave-tcgx"
server_key: "0d6229970bb03f7ff78a6f1b1f857990"
server_key_iv: "73ad8e1040014925d0e1defae4b3a146"
server_key_version: "encryptionkey@0"
I20260620 23:40:38.297380 19256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998525501163-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998525501163-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998525501163-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:40:38.317008 19256 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:40:38.317939 19256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:40:38.318236 19256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:40:38.319366 19256 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:40:38.320917 19256 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:40:38.321045 19256 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:38.321177 19256 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:40:38.321261 19256 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:38.355643 19256 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40905
I20260620 23:40:38.355737 19382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40905 every 8 connection(s)
I20260620 23:40:38.357034 19256 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
I20260620 23:40:38.357512 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 19256
I20260620 23:40:38.357692 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998525501163-388-0/minicluster-data/ts-0/wal/instance
I20260620 23:40:38.358290 388 external_mini_cluster.cc:1428] Setting key 274803bd219a1555dda0453135af53ba
I20260620 23:40:38.361500 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998525501163-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998525501163-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998525501163-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39527
--builtin_ntp_servers=127.0.97.20:35429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 23:40:38.371667 19385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:38.360025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39831 (local address 127.0.97.62:39527)
0620 23:40:38.360505 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:38.360511 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:38.361961 (+ 1450us) server_negotiation.cc:407] Connection header received
0620 23:40:38.363866 (+ 1905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:38.363875 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:38.363970 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:38.364096 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:40:38.365174 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:38.366406 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:38.368190 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:38.368948 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:38.369073 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:38.370839 (+ 1766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:40:38.370844 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:40:38.370847 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:40:38.370867 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:40:38.370915 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:40:38.370927 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:40:38.371073 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:38.371319 (+ 246us) server_negotiation.cc:299] Negotiation successful
0620 23:40:38.371411 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":228,"threads_started":1}
I20260620 23:40:38.373224 19386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:38.360693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39527 (local address 127.0.97.1:39831)
0620 23:40:38.361788 (+ 1095us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:38.361834 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 23:40:38.363593 (+ 1759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:40:38.364290 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:40:38.364299 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:40:38.364389 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:40:38.364987 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:38.365005 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:38.366565 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:38.366569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:38.368033 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:38.368045 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:38.369209 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:38.369219 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:38.369565 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:38.370449 (+ 884us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:40:38.370455 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:40:38.370515 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:40:38.370633 (+ 118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:40:38.371084 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:40:38.371098 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:40:38.371109 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 23:40:38.371304 (+ 195us) client_negotiation.cc:240] Negotiation successful
0620 23:40:38.371762 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":360,"threads_started":1}
I20260620 23:40:38.374826 19383 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39527
I20260620 23:40:38.375176 19383 heartbeater.cc:464] Registering TS with master...
I20260620 23:40:38.376003 19383 heartbeater.cc:511] Master 127.0.97.62:39527 requested a full tablet report, sending...
I20260620 23:40:38.378214 19197 ts_manager.cc:194] Registered new tserver with Master: c84eda7f94864d5d86f4fba4a485f01b (127.0.97.1:40905)
I20260620 23:40:38.379769 19197 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:39831
W20260620 23:40:38.710805 19387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:40:38.711211 19387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:40:38.711380 19387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:40:38.719098 19387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:40:38.719242 19387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:40:38.719430 19387 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 23:40:38.729147 19387 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998525501163-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998525501163-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39527
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998525501163-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:40:38.730607 19387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:40:38.732589 19387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:40:38.746500 19392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:40:38.746882 19395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:40:38.746917 19393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:40:38.747525 19387 server_base.cc:1034] running on GCE node
I20260620 23:40:38.748134 19387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:40:38.749163 19387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:40:38.750429 19387 hybrid_clock.cc:648] HybridClock initialized: now 1781998838750377 us; error 44 us; skew 500 ppm
I20260620 23:40:38.750773 19387 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:40:38.753130 19387 webserver.cc:460] Webserver started at http://127.0.97.2:46121/ using document root <none> and password file <none>
I20260620 23:40:38.753859 19387 fs_manager.cc:359] Metadata directory not provided
I20260620 23:40:38.753995 19387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:40:38.754329 19387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:40:38.757510 19387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998525501163-388-0/minicluster-data/ts-1/data/instance:
uuid: "b04e2d87b42d437f8735a536e896c858"
format_stamp: "Formatted at 2026-06-20 23:40:38 on dist-test-slave-tcgx"
server_key: "bb736445cf9fe93888564d2e52102acc"
server_key_iv: "88efec789f756af9db1a2030e5c1d35d"
server_key_version: "encryptionkey@0"
I20260620 23:40:38.758320 19387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998525501163-388-0/minicluster-data/ts-1/wal/instance:
uuid: "b04e2d87b42d437f8735a536e896c858"
format_stamp: "Formatted at 2026-06-20 23:40:38 on dist-test-slave-tcgx"
server_key: "bb736445cf9fe93888564d2e52102acc"
server_key_iv: "88efec789f756af9db1a2030e5c1d35d"
server_key_version: "encryptionkey@0"
I20260620 23:40:38.763667 19387 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 23:40:38.767409 19401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:38.769239 19387 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 23:40:38.769454 19387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998525501163-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998525501163-388-0/minicluster-data/ts-1/wal
uuid: "b04e2d87b42d437f8735a536e896c858"
format_stamp: "Formatted at 2026-06-20 23:40:38 on dist-test-slave-tcgx"
server_key: "bb736445cf9fe93888564d2e52102acc"
server_key_iv: "88efec789f756af9db1a2030e5c1d35d"
server_key_version: "encryptionkey@0"
I20260620 23:40:38.769735 19387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998525501163-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998525501163-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998525501163-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:40:38.803630 19387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:40:38.804564 19387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:40:38.804894 19387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:40:38.805995 19387 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:40:38.807672 19387 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:40:38.807830 19387 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:38.807957 19387 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:40:38.808046 19387 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:38.842096 19387 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:32933
I20260620 23:40:38.842185 19513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:32933 every 8 connection(s)
I20260620 23:40:38.843500 19387 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
I20260620 23:40:38.850723 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 19387
I20260620 23:40:38.850899 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998525501163-388-0/minicluster-data/ts-1/wal/instance
I20260620 23:40:38.851452 388 external_mini_cluster.cc:1428] Setting key 91594e6fe5b5c312a27c6704783a00e6
I20260620 23:40:38.854357 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998525501163-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998525501163-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998525501163-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39527
--builtin_ntp_servers=127.0.97.20:35429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 23:40:38.857177 19385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:38.846240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59663 (local address 127.0.97.62:39527)
0620 23:40:38.846427 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:38.846432 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:38.848040 (+ 1608us) server_negotiation.cc:407] Connection header received
0620 23:40:38.849911 (+ 1871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:38.849916 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:38.849982 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:38.850071 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:40:38.851257 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:38.852254 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:38.854017 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:38.854687 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:38.854818 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:38.856272 (+ 1454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:40:38.856281 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:40:38.856288 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:40:38.856313 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:40:38.856358 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:40:38.856376 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:40:38.856470 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:38.856689 (+ 219us) server_negotiation.cc:299] Negotiation successful
0620 23:40:38.856798 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 23:40:38.858078 19516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:38.846820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39527 (local address 127.0.97.2:59663)
0620 23:40:38.847836 (+ 1016us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:38.847875 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 23:40:38.849641 (+ 1766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:40:38.850284 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:40:38.850292 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:40:38.850384 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:40:38.851080 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:38.851101 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:38.852420 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:38.852424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:38.853862 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:38.853876 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:38.854920 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:38.854926 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:38.855205 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:38.855957 (+ 752us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:40:38.855959 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:40:38.856000 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:40:38.856080 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:40:38.856472 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:40:38.856480 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:40:38.856485 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 23:40:38.856630 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 23:40:38.856944 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":290,"threads_started":1}
I20260620 23:40:38.859696 19514 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39527
I20260620 23:40:38.860025 19514 heartbeater.cc:464] Registering TS with master...
I20260620 23:40:38.860944 19514 heartbeater.cc:511] Master 127.0.97.62:39527 requested a full tablet report, sending...
I20260620 23:40:38.862525 19197 ts_manager.cc:194] Registered new tserver with Master: b04e2d87b42d437f8735a536e896c858 (127.0.97.2:32933)
I20260620 23:40:38.863447 19197 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:59663
W20260620 23:40:39.190829 19517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:40:39.191205 19517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:40:39.191345 19517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:40:39.198698 19517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:40:39.198814 19517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:40:39.199026 19517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 23:40:39.208690 19517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998525501163-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998525501163-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39527
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998525501163-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:40:39.210117 19517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:40:39.211985 19517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:40:39.225847 19525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:40:39.225935 19523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:40:39.225889 19522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:40:39.226820 19517 server_base.cc:1034] running on GCE node
I20260620 23:40:39.227489 19517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:40:39.228480 19517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:40:39.229683 19517 hybrid_clock.cc:648] HybridClock initialized: now 1781998839229643 us; error 32 us; skew 500 ppm
I20260620 23:40:39.230031 19517 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:40:39.232383 19517 webserver.cc:460] Webserver started at http://127.0.97.3:44447/ using document root <none> and password file <none>
I20260620 23:40:39.233104 19517 fs_manager.cc:359] Metadata directory not provided
I20260620 23:40:39.233214 19517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:40:39.233548 19517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:40:39.236635 19517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998525501163-388-0/minicluster-data/ts-2/data/instance:
uuid: "492aa9c3763c4c5d8b624a519377d16d"
format_stamp: "Formatted at 2026-06-20 23:40:39 on dist-test-slave-tcgx"
server_key: "ca7f1858494678db93ab31c281777b0c"
server_key_iv: "2b94e5cb6c4dc96c94513c16d1559d7e"
server_key_version: "encryptionkey@0"
I20260620 23:40:39.237459 19517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998525501163-388-0/minicluster-data/ts-2/wal/instance:
uuid: "492aa9c3763c4c5d8b624a519377d16d"
format_stamp: "Formatted at 2026-06-20 23:40:39 on dist-test-slave-tcgx"
server_key: "ca7f1858494678db93ab31c281777b0c"
server_key_iv: "2b94e5cb6c4dc96c94513c16d1559d7e"
server_key_version: "encryptionkey@0"
I20260620 23:40:39.242756 19517 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 23:40:39.246539 19531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:39.248158 19517 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:40:39.248327 19517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998525501163-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998525501163-388-0/minicluster-data/ts-2/wal
uuid: "492aa9c3763c4c5d8b624a519377d16d"
format_stamp: "Formatted at 2026-06-20 23:40:39 on dist-test-slave-tcgx"
server_key: "ca7f1858494678db93ab31c281777b0c"
server_key_iv: "2b94e5cb6c4dc96c94513c16d1559d7e"
server_key_version: "encryptionkey@0"
I20260620 23:40:39.248543 19517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998525501163-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998525501163-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998525501163-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:40:39.276873 19517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:40:39.277786 19517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:40:39.278079 19517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:40:39.279143 19517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:40:39.280674 19517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:40:39.280761 19517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:39.280925 19517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:40:39.281006 19517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:39.315799 19517 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43747
I20260620 23:40:39.315896 19643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43747 every 8 connection(s)
I20260620 23:40:39.317168 19517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
I20260620 23:40:39.322284 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 19517
I20260620 23:40:39.322458 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998525501163-388-0/minicluster-data/ts-2/wal/instance
I20260620 23:40:39.322897 388 external_mini_cluster.cc:1428] Setting key e0553272636c52f1b9811be8ab5d5126
I20260620 23:40:39.330264 19385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:39.319948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46933 (local address 127.0.97.62:39527)
0620 23:40:39.320130 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:39.320134 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:39.321465 (+ 1331us) server_negotiation.cc:407] Connection header received
0620 23:40:39.323446 (+ 1981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:39.323452 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:39.323551 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:39.323662 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:40:39.325149 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:39.325994 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:39.327672 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:39.328103 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:39.328198 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:39.329531 (+ 1333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:40:39.329535 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:40:39.329538 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:40:39.329557 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:40:39.329585 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:40:39.329595 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:40:39.329659 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:39.329942 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 23:40:39.330021 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 23:40:39.331422 19646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:39.320393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39527 (local address 127.0.97.3:46933)
0620 23:40:39.321304 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:39.321340 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 23:40:39.323164 (+ 1824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:40:39.323941 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:40:39.323955 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:40:39.324109 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:40:39.324968 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:39.324995 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:39.326118 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:39.326121 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:39.327540 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:39.327552 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:39.328288 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:39.328292 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:39.328495 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:39.329231 (+ 736us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:40:39.329234 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:40:39.329282 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:40:39.329364 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:40:39.329674 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:40:39.329681 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:40:39.329687 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 23:40:39.329921 (+ 234us) client_negotiation.cc:240] Negotiation successful
0620 23:40:39.330230 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":285,"threads_started":1}
I20260620 23:40:39.332911 19644 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39527
I20260620 23:40:39.333197 19644 heartbeater.cc:464] Registering TS with master...
I20260620 23:40:39.333972 19644 heartbeater.cc:511] Master 127.0.97.62:39527 requested a full tablet report, sending...
I20260620 23:40:39.335563 19197 ts_manager.cc:194] Registered new tserver with Master: 492aa9c3763c4c5d8b624a519377d16d (127.0.97.3:43747)
I20260620 23:40:39.336539 19197 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:46933
I20260620 23:40:39.338347 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:40:39.349299 19385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:39.343005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44064 (local address 127.0.97.62:39527)
0620 23:40:39.343277 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:39.343283 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:39.343542 (+ 259us) server_negotiation.cc:407] Connection header received
0620 23:40:39.343682 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:39.343686 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:39.343769 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:39.343856 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:40:39.344351 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:39.345052 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:39.346513 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:39.346999 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:39.347101 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:39.348478 (+ 1377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:40:39.348482 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:40:39.348486 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:40:39.348505 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:40:39.348539 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:40:39.348559 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:40:39.348682 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:39.348855 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 23:40:39.348948 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 23:40:39.353897 19196 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44064:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:40:39.363669 19196 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:40:39.383201 19383 heartbeater.cc:503] Master 127.0.97.62:39527 was elected leader, sending a full tablet report...
I20260620 23:40:39.399228 19657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:39.376311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43747 (local address 127.0.0.1:60828)
0620 23:40:39.388081 (+ 11770us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:39.389465 (+ 1384us) client_negotiation.cc:174] Beginning negotiation
0620 23:40:39.389575 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:40:39.390873 (+ 1298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:40:39.390879 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:40:39.390911 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:40:39.391394 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:39.391410 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:39.393529 (+ 2119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:39.393535 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:39.396185 (+ 2650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:39.396198 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:39.398062 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:39.398070 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:39.398306 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:39.398380 (+ 74us) client_negotiation.cc:769] Sending connection context
0620 23:40:39.398554 (+ 174us) client_negotiation.cc:240] Negotiation successful
0620 23:40:39.398754 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11364,"mutex_wait_us":103,"thread_start_us":11770,"threads_started":1}
I20260620 23:40:39.399212 19658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:39.376517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60828 (local address 127.0.97.3:43747)
0620 23:40:39.390190 (+ 13673us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:39.390214 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:39.390255 (+ 41us) server_negotiation.cc:407] Connection header received
0620 23:40:39.390364 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:39.390375 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:39.390643 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:39.391610 (+ 967us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:40:39.391893 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:39.393373 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:39.396443 (+ 3070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:39.397836 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:39.397991 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:39.398293 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:39.398629 (+ 336us) server_negotiation.cc:299] Negotiation successful
0620 23:40:39.398799 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13463,"thread_start_us":13300,"threads_started":1}
I20260620 23:40:39.399340 19656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:39.376340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40905 (local address 127.0.0.1:43650)
0620 23:40:39.388083 (+ 11743us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:39.388150 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0620 23:40:39.388371 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:40:39.389832 (+ 1461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:40:39.389840 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:40:39.389876 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:40:39.390316 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:39.390331 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:39.394015 (+ 3684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:39.394021 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:39.396507 (+ 2486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:39.396522 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:39.398448 (+ 1926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:39.398455 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:39.398659 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:39.398692 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 23:40:39.398777 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 23:40:39.398926 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11636,"mutex_wait_us":54,"thread_start_us":11191,"threads_started":1}
I20260620 23:40:39.400032 19659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:39.376694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43650 (local address 127.0.97.1:40905)
0620 23:40:39.385831 (+ 9137us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:39.385842 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:39.388687 (+ 2845us) server_negotiation.cc:407] Connection header received
0620 23:40:39.388806 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:39.388825 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:39.389110 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:39.389318 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:40:39.390736 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:39.393857 (+ 3121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:39.396736 (+ 2879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:39.397990 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:39.398135 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:39.398405 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:39.399481 (+ 1076us) server_negotiation.cc:299] Negotiation successful
0620 23:40:39.399680 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8912,"thread_start_us":8764,"threads_started":1}
I20260620 23:40:39.403100 19661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:39.390359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:32933 (local address 127.0.0.1:35052)
0620 23:40:39.393992 (+ 3633us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:39.394012 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:40:39.394174 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:40:39.394909 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:40:39.394914 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:40:39.394966 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:40:39.395476 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:39.395493 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:39.397397 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:39.397402 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:39.399575 (+ 2173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:39.399589 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:39.402192 (+ 2603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:39.402201 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:39.402397 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:39.402431 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 23:40:39.402519 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 23:40:39.402652 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3479,"mutex_wait_us":5,"thread_start_us":3314,"threads_started":1}
I20260620 23:40:39.403167 19660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:39.389420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35052 (local address 127.0.97.2:32933)
0620 23:40:39.393580 (+ 4160us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:39.393593 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:39.394145 (+ 552us) server_negotiation.cc:407] Connection header received
0620 23:40:39.394360 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:39.394370 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:39.394674 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:39.394908 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:40:39.395717 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:39.397237 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:39.399816 (+ 2579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:39.401661 (+ 1845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:39.401817 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:39.402056 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:39.402616 (+ 560us) server_negotiation.cc:299] Negotiation successful
0620 23:40:39.402777 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4001,"thread_start_us":3867,"threads_started":1}
I20260620 23:40:39.404102 19318 tablet_service.cc:1467] Processing CreateTablet for tablet 05a5fea078a741bba0813a5dbbcd8d26 (DEFAULT_TABLE table=test-table [id=1bb16e4d5d7c453994309719375133b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:40:39.404102 19579 tablet_service.cc:1467] Processing CreateTablet for tablet 05a5fea078a741bba0813a5dbbcd8d26 (DEFAULT_TABLE table=test-table [id=1bb16e4d5d7c453994309719375133b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:40:39.405376 19318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05a5fea078a741bba0813a5dbbcd8d26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:40:39.405372 19579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05a5fea078a741bba0813a5dbbcd8d26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:40:39.410809 19449 tablet_service.cc:1467] Processing CreateTablet for tablet 05a5fea078a741bba0813a5dbbcd8d26 (DEFAULT_TABLE table=test-table [id=1bb16e4d5d7c453994309719375133b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:40:39.413398 19449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05a5fea078a741bba0813a5dbbcd8d26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:40:39.417976 19662 tablet_bootstrap.cc:492] T 05a5fea078a741bba0813a5dbbcd8d26 P 492aa9c3763c4c5d8b624a519377d16d: Bootstrap starting.
I20260620 23:40:39.421877 19662 tablet_bootstrap.cc:654] T 05a5fea078a741bba0813a5dbbcd8d26 P 492aa9c3763c4c5d8b624a519377d16d: Neither blocks nor log segments found. Creating new log.
I20260620 23:40:39.423058 19662 log.cc:824] T 05a5fea078a741bba0813a5dbbcd8d26 P 492aa9c3763c4c5d8b624a519377d16d: Log is configured to *not* fsync() on all Append() calls
I20260620 23:40:39.423784 19663 tablet_bootstrap.cc:492] T 05a5fea078a741bba0813a5dbbcd8d26 P c84eda7f94864d5d86f4fba4a485f01b: Bootstrap starting.
I20260620 23:40:39.428020 19663 tablet_bootstrap.cc:654] T 05a5fea078a741bba0813a5dbbcd8d26 P c84eda7f94864d5d86f4fba4a485f01b: Neither blocks nor log segments found. Creating new log.
I20260620 23:40:39.428666 19662 tablet_bootstrap.cc:492] T 05a5fea078a741bba0813a5dbbcd8d26 P 492aa9c3763c4c5d8b624a519377d16d: No bootstrap required, opened a new log
I20260620 23:40:39.428969 19662 ts_tablet_manager.cc:1397] T 05a5fea078a741bba0813a5dbbcd8d26 P 492aa9c3763c4c5d8b624a519377d16d: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260620 23:40:39.429332 19663 log.cc:824] T 05a5fea078a741bba0813a5dbbcd8d26 P c84eda7f94864d5d86f4fba4a485f01b: Log is configured to *not* fsync() on all Append() calls
I20260620 23:40:39.431514 19666 tablet_bootstrap.cc:492] T 05a5fea078a741bba0813a5dbbcd8d26 P b04e2d87b42d437f8735a536e896c858: Bootstrap starting.
I20260620 23:40:39.432710 19663 tablet_bootstrap.cc:492] T 05a5fea078a741bba0813a5dbbcd8d26 P c84eda7f94864d5d86f4fba4a485f01b: No bootstrap required, opened a new log
I20260620 23:40:39.432987 19663 ts_tablet_manager.cc:1397] T 05a5fea078a741bba0813a5dbbcd8d26 P c84eda7f94864d5d86f4fba4a485f01b: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260620 23:40:39.434652 19666 tablet_bootstrap.cc:654] T 05a5fea078a741bba0813a5dbbcd8d26 P b04e2d87b42d437f8735a536e896c858: Neither blocks nor log segments found. Creating new log.
I20260620 23:40:39.435850 19666 log.cc:824] T 05a5fea078a741bba0813a5dbbcd8d26 P b04e2d87b42d437f8735a536e896c858: Log is configured to *not* fsync() on all Append() calls
I20260620 23:40:39.438601 19666 tablet_bootstrap.cc:492] T 05a5fea078a741bba0813a5dbbcd8d26 P b04e2d87b42d437f8735a536e896c858: No bootstrap required, opened a new log
I20260620 23:40:39.438872 19666 ts_tablet_manager.cc:1397] T 05a5fea078a741bba0813a5dbbcd8d26 P b04e2d87b42d437f8735a536e896c858: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260620 23:40:39.440284 19662 raft_consensus.cc:348] T 05a5fea078a741bba0813a5dbbcd8d26 P 492aa9c3763c4c5d8b624a519377d16d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492aa9c3763c4c5d8b624a519377d16d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43747 } } peers { permanent_uuid: "c84eda7f94864d5d86f4fba4a485f01b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40905 } } peers { permanent_uuid: "b04e2d87b42d437f8735a536e896c858" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32933 } }
I20260620 23:40:39.441011 19662 raft_consensus.cc:374] T 05a5fea078a741bba0813a5dbbcd8d26 P 492aa9c3763c4c5d8b624a519377d16d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:40:39.441161 19662 raft_consensus.cc:729] T 05a5fea078a741bba0813a5dbbcd8d26 P 492aa9c3763c4c5d8b624a519377d16d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492aa9c3763c4c5d8b624a519377d16d, State: Initialized, Role: FOLLOWER
I20260620 23:40:39.442030 19662 consensus_queue.cc:260] T 05a5fea078a741bba0813a5dbbcd8d26 P 492aa9c3763c4c5d8b624a519377d16d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492aa9c3763c4c5d8b624a519377d16d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43747 } } peers { permanent_uuid: "c84eda7f94864d5d86f4fba4a485f01b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40905 } } peers { permanent_uuid: "b04e2d87b42d437f8735a536e896c858" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32933 } }
I20260620 23:40:39.443316 19644 heartbeater.cc:503] Master 127.0.97.62:39527 was elected leader, sending a full tablet report...
I20260620 23:40:39.444177 19663 raft_consensus.cc:348] T 05a5fea078a741bba0813a5dbbcd8d26 P c84eda7f94864d5d86f4fba4a485f01b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492aa9c3763c4c5d8b624a519377d16d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43747 } } peers { permanent_uuid: "c84eda7f94864d5d86f4fba4a485f01b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40905 } } peers { permanent_uuid: "b04e2d87b42d437f8735a536e896c858" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32933 } }
I20260620 23:40:39.444715 19663 raft_consensus.cc:374] T 05a5fea078a741bba0813a5dbbcd8d26 P c84eda7f94864d5d86f4fba4a485f01b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:40:39.444851 19663 raft_consensus.cc:729] T 05a5fea078a741bba0813a5dbbcd8d26 P c84eda7f94864d5d86f4fba4a485f01b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c84eda7f94864d5d86f4fba4a485f01b, State: Initialized, Role: FOLLOWER
I20260620 23:40:39.445598 19662 ts_tablet_manager.cc:1428] T 05a5fea078a741bba0813a5dbbcd8d26 P 492aa9c3763c4c5d8b624a519377d16d: Time spent starting tablet: real 0.016s user 0.010s sys 0.004s
I20260620 23:40:39.445580 19663 consensus_queue.cc:260] T 05a5fea078a741bba0813a5dbbcd8d26 P c84eda7f94864d5d86f4fba4a485f01b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492aa9c3763c4c5d8b624a519377d16d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43747 } } peers { permanent_uuid: "c84eda7f94864d5d86f4fba4a485f01b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40905 } } peers { permanent_uuid: "b04e2d87b42d437f8735a536e896c858" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32933 } }
I20260620 23:40:39.448545 19663 ts_tablet_manager.cc:1428] T 05a5fea078a741bba0813a5dbbcd8d26 P c84eda7f94864d5d86f4fba4a485f01b: Time spent starting tablet: real 0.015s user 0.006s sys 0.009s
I20260620 23:40:39.448678 19666 raft_consensus.cc:348] T 05a5fea078a741bba0813a5dbbcd8d26 P b04e2d87b42d437f8735a536e896c858 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492aa9c3763c4c5d8b624a519377d16d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43747 } } peers { permanent_uuid: "c84eda7f94864d5d86f4fba4a485f01b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40905 } } peers { permanent_uuid: "b04e2d87b42d437f8735a536e896c858" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32933 } }
I20260620 23:40:39.449194 19666 raft_consensus.cc:374] T 05a5fea078a741bba0813a5dbbcd8d26 P b04e2d87b42d437f8735a536e896c858 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:40:39.449309 19666 raft_consensus.cc:729] T 05a5fea078a741bba0813a5dbbcd8d26 P b04e2d87b42d437f8735a536e896c858 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b04e2d87b42d437f8735a536e896c858, State: Initialized, Role: FOLLOWER
I20260620 23:40:39.450136 19666 consensus_queue.cc:260] T 05a5fea078a741bba0813a5dbbcd8d26 P b04e2d87b42d437f8735a536e896c858 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492aa9c3763c4c5d8b624a519377d16d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43747 } } peers { permanent_uuid: "c84eda7f94864d5d86f4fba4a485f01b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40905 } } peers { permanent_uuid: "b04e2d87b42d437f8735a536e896c858" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32933 } }
I20260620 23:40:39.451327 19514 heartbeater.cc:503] Master 127.0.97.62:39527 was elected leader, sending a full tablet report...
I20260620 23:40:39.452001 19666 ts_tablet_manager.cc:1428] T 05a5fea078a741bba0813a5dbbcd8d26 P b04e2d87b42d437f8735a536e896c858: Time spent starting tablet: real 0.013s user 0.007s sys 0.005s
I20260620 23:40:39.546082 19669 raft_consensus.cc:482] T 05a5fea078a741bba0813a5dbbcd8d26 P c84eda7f94864d5d86f4fba4a485f01b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:40:39.546480 19669 raft_consensus.cc:504] T 05a5fea078a741bba0813a5dbbcd8d26 P c84eda7f94864d5d86f4fba4a485f01b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492aa9c3763c4c5d8b624a519377d16d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43747 } } peers { permanent_uuid: "c84eda7f94864d5d86f4fba4a485f01b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40905 } } peers { permanent_uuid: "b04e2d87b42d437f8735a536e896c858" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32933 } }
I20260620 23:40:39.548321 19669 leader_election.cc:288] T 05a5fea078a741bba0813a5dbbcd8d26 P c84eda7f94864d5d86f4fba4a485f01b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 492aa9c3763c4c5d8b624a519377d16d (127.0.97.3:43747), b04e2d87b42d437f8735a536e896c858 (127.0.97.2:32933)
I20260620 23:40:39.554651 19668 raft_consensus.cc:482] T 05a5fea078a741bba0813a5dbbcd8d26 P 492aa9c3763c4c5d8b624a519377d16d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:40:39.555058 19668 raft_consensus.cc:504] T 05a5fea078a741bba0813a5dbbcd8d26 P 492aa9c3763c4c5d8b624a519377d16d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492aa9c3763c4c5d8b624a519377d16d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43747 } } peers { permanent_uuid: "c84eda7f94864d5d86f4fba4a485f01b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40905 } } peers { permanent_uuid: "b04e2d87b42d437f8735a536e896c858" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32933 } }
I20260620 23:40:39.555775 19660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:39.548850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33705 (local address 127.0.97.2:32933)
0620 23:40:39.549043 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:39.549050 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:39.550195 (+ 1145us) server_negotiation.cc:407] Connection header received
0620 23:40:39.550403 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:39.550408 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:39.550490 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:39.550591 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:40:39.551155 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:39.551848 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:39.553830 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:39.554701 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:39.554856 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:39.554992 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:39.555341 (+ 349us) server_negotiation.cc:299] Negotiation successful
0620 23:40:39.555489 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 23:40:39.555838 19671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:39.548817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:32933 (local address 127.0.97.1:33705)
0620 23:40:39.550075 (+ 1258us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:39.550089 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:40:39.550244 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:40:39.550608 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:40:39.550612 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:40:39.550634 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:40:39.550955 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:39.550985 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:39.552001 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:39.552009 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:39.553641 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:39.553653 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:39.554915 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:39.554921 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:39.555140 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:39.555182 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 23:40:39.555278 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 23:40:39.555433 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1147,"mutex_wait_us":31,"thread_start_us":853,"threads_started":1}
I20260620 23:40:39.555908 19658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:39.548843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47725 (local address 127.0.97.3:43747)
0620 23:40:39.549043 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:39.549050 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:39.549641 (+ 591us) server_negotiation.cc:407] Connection header received
0620 23:40:39.549909 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:39.549915 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:39.549992 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:39.550094 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:40:39.550687 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:39.551531 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:39.553662 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:39.554716 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:39.554883 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:39.555009 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:39.555502 (+ 493us) server_negotiation.cc:299] Negotiation successful
0620 23:40:39.555665 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 23:40:39.556648 19672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:39.548817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43747 (local address 127.0.97.1:47725)
0620 23:40:39.549514 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:39.549530 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:40:39.549721 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:40:39.550111 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:40:39.550116 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:40:39.550139 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:40:39.550543 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:39.550552 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:39.551663 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:39.551671 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:39.553451 (+ 1780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:39.553464 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:39.555140 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:39.555146 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:39.555321 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:39.555349 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 23:40:39.556121 (+ 772us) client_negotiation.cc:240] Negotiation successful
0620 23:40:39.556296 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"mutex_wait_us":72,"spinlock_wait_cycles":39680,"thread_start_us":430,"threads_started":1}
I20260620 23:40:39.557273 19668 leader_election.cc:288] T 05a5fea078a741bba0813a5dbbcd8d26 P 492aa9c3763c4c5d8b624a519377d16d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c84eda7f94864d5d86f4fba4a485f01b (127.0.97.1:40905), b04e2d87b42d437f8735a536e896c858 (127.0.97.2:32933)
I20260620 23:40:39.557102 19469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a5fea078a741bba0813a5dbbcd8d26" candidate_uuid: "c84eda7f94864d5d86f4fba4a485f01b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b04e2d87b42d437f8735a536e896c858" is_pre_election: true
I20260620 23:40:39.557672 19469 raft_consensus.cc:2454] T 05a5fea078a741bba0813a5dbbcd8d26 P b04e2d87b42d437f8735a536e896c858 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c84eda7f94864d5d86f4fba4a485f01b in term 0.
I20260620 23:40:39.557775 19599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a5fea078a741bba0813a5dbbcd8d26" candidate_uuid: "c84eda7f94864d5d86f4fba4a485f01b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492aa9c3763c4c5d8b624a519377d16d" is_pre_election: true
I20260620 23:40:39.558248 19599 raft_consensus.cc:2454] T 05a5fea078a741bba0813a5dbbcd8d26 P 492aa9c3763c4c5d8b624a519377d16d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c84eda7f94864d5d86f4fba4a485f01b in term 0.
I20260620 23:40:39.559449 19271 leader_election.cc:302] T 05a5fea078a741bba0813a5dbbcd8d26 P c84eda7f94864d5d86f4fba4a485f01b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b04e2d87b42d437f8735a536e896c858, c84eda7f94864d5d86f4fba4a485f01b; no voters:
I20260620 23:40:39.560163 19669 raft_consensus.cc:2781] T 05a5fea078a741bba0813a5dbbcd8d26 P c84eda7f94864d5d86f4fba4a485f01b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:40:39.560335 19669 raft_consensus.cc:482] T 05a5fea078a741bba0813a5dbbcd8d26 P c84eda7f94864d5d86f4fba4a485f01b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:40:39.560460 19669 raft_consensus.cc:3037] T 05a5fea078a741bba0813a5dbbcd8d26 P c84eda7f94864d5d86f4fba4a485f01b [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:40:39.563784 19669 raft_consensus.cc:504] T 05a5fea078a741bba0813a5dbbcd8d26 P c84eda7f94864d5d86f4fba4a485f01b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492aa9c3763c4c5d8b624a519377d16d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43747 } } peers { permanent_uuid: "c84eda7f94864d5d86f4fba4a485f01b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40905 } } peers { permanent_uuid: "b04e2d87b42d437f8735a536e896c858" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32933 } }
I20260620 23:40:39.564651 19659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:39.558154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56697 (local address 127.0.97.1:40905)
0620 23:40:39.558334 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:39.558338 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:39.558352 (+ 14us) server_negotiation.cc:407] Connection header received
0620 23:40:39.558402 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:39.558405 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:39.558519 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:39.558623 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:40:39.559142 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:39.560209 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:39.562674 (+ 2465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:39.563691 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:39.563847 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:39.563959 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:39.564290 (+ 331us) server_negotiation.cc:299] Negotiation successful
0620 23:40:39.564420 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 23:40:39.564662 19646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:39.557542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40905 (local address 127.0.97.3:56697)
0620 23:40:39.557792 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:39.557815 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 23:40:39.557933 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:40:39.558652 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:40:39.558658 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:40:39.558687 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:40:39.558911 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:39.558923 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:39.560386 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:39.560391 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:39.562489 (+ 2098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:39.562507 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:39.563857 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:39.563863 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:39.564077 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:39.564110 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 23:40:39.564228 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 23:40:39.564374 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108}
I20260620 23:40:39.565138 19599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a5fea078a741bba0813a5dbbcd8d26" candidate_uuid: "c84eda7f94864d5d86f4fba4a485f01b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "492aa9c3763c4c5d8b624a519377d16d"
I20260620 23:40:39.565369 19599 raft_consensus.cc:3037] T 05a5fea078a741bba0813a5dbbcd8d26 P 492aa9c3763c4c5d8b624a519377d16d [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:40:39.565521 19673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:39.557983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:32933 (local address 127.0.97.3:58917)
0620 23:40:39.559295 (+ 1312us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:39.559313 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:40:39.559422 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:40:39.559967 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:40:39.559971 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:40:39.559995 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:40:39.560373 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:39.560386 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:39.561422 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:39.561427 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:39.562995 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:39.563010 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:39.564784 (+ 1774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:39.564790 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:39.564971 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:39.565010 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 23:40:39.565089 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 23:40:39.565230 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1188,"thread_start_us":1003,"threads_started":1}
I20260620 23:40:39.565726 19338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a5fea078a741bba0813a5dbbcd8d26" candidate_uuid: "492aa9c3763c4c5d8b624a519377d16d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c84eda7f94864d5d86f4fba4a485f01b" is_pre_election: true
I20260620 23:40:39.566145 19660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:39.558990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58917 (local address 127.0.97.2:32933)
0620 23:40:39.559682 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:39.559686 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:39.559698 (+ 12us) server_negotiation.cc:407] Connection header received
0620 23:40:39.559770 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:39.559775 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:39.559845 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:39.559955 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:40:39.560564 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:39.561267 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:39.563201 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:39.564436 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:39.564573 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:39.564676 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:39.565717 (+ 1041us) server_negotiation.cc:299] Negotiation successful
0620 23:40:39.565884 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559}
I20260620 23:40:39.566156 19338 raft_consensus.cc:2379] T 05a5fea078a741bba0813a5dbbcd8d26 P c84eda7f94864d5d86f4fba4a485f01b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 492aa9c3763c4c5d8b624a519377d16d in current term 1: Already voted for candidate c84eda7f94864d5d86f4fba4a485f01b in this term.
I20260620 23:40:39.567730 19669 leader_election.cc:288] T 05a5fea078a741bba0813a5dbbcd8d26 P c84eda7f94864d5d86f4fba4a485f01b [CANDIDATE]: Term 1 election: Requested vote from peers 492aa9c3763c4c5d8b624a519377d16d (127.0.97.3:43747), b04e2d87b42d437f8735a536e896c858 (127.0.97.2:32933)
I20260620 23:40:39.568048 19599 raft_consensus.cc:2454] T 05a5fea078a741bba0813a5dbbcd8d26 P 492aa9c3763c4c5d8b624a519377d16d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c84eda7f94864d5d86f4fba4a485f01b in term 1.
I20260620 23:40:39.568107 19469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a5fea078a741bba0813a5dbbcd8d26" candidate_uuid: "492aa9c3763c4c5d8b624a519377d16d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b04e2d87b42d437f8735a536e896c858" is_pre_election: true
I20260620 23:40:39.568354 19469 raft_consensus.cc:2454] T 05a5fea078a741bba0813a5dbbcd8d26 P b04e2d87b42d437f8735a536e896c858 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492aa9c3763c4c5d8b624a519377d16d in term 0.
I20260620 23:40:39.568454 19272 leader_election.cc:302] T 05a5fea078a741bba0813a5dbbcd8d26 P c84eda7f94864d5d86f4fba4a485f01b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492aa9c3763c4c5d8b624a519377d16d, c84eda7f94864d5d86f4fba4a485f01b; no voters:
I20260620 23:40:39.568310 19468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a5fea078a741bba0813a5dbbcd8d26" candidate_uuid: "c84eda7f94864d5d86f4fba4a485f01b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b04e2d87b42d437f8735a536e896c858"
I20260620 23:40:39.568647 19468 raft_consensus.cc:3037] T 05a5fea078a741bba0813a5dbbcd8d26 P b04e2d87b42d437f8735a536e896c858 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:40:39.568841 19669 raft_consensus.cc:2781] T 05a5fea078a741bba0813a5dbbcd8d26 P c84eda7f94864d5d86f4fba4a485f01b [term 1 FOLLOWER]: Leader election won for term 1
W20260620 23:40:39.568990 19645 tablet.cc:2369] T 05a5fea078a741bba0813a5dbbcd8d26 P 492aa9c3763c4c5d8b624a519377d16d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:40:39.570207 19669 raft_consensus.cc:686] T 05a5fea078a741bba0813a5dbbcd8d26 P c84eda7f94864d5d86f4fba4a485f01b [term 1 LEADER]: Becoming Leader. State: Replica: c84eda7f94864d5d86f4fba4a485f01b, State: Running, Role: LEADER
I20260620 23:40:39.570139 19532 leader_election.cc:302] T 05a5fea078a741bba0813a5dbbcd8d26 P 492aa9c3763c4c5d8b624a519377d16d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 492aa9c3763c4c5d8b624a519377d16d, b04e2d87b42d437f8735a536e896c858; no voters: c84eda7f94864d5d86f4fba4a485f01b
I20260620 23:40:39.570858 19668 raft_consensus.cc:2741] T 05a5fea078a741bba0813a5dbbcd8d26 P 492aa9c3763c4c5d8b624a519377d16d [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260620 23:40:39.570807 19669 consensus_queue.cc:237] T 05a5fea078a741bba0813a5dbbcd8d26 P c84eda7f94864d5d86f4fba4a485f01b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492aa9c3763c4c5d8b624a519377d16d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43747 } } peers { permanent_uuid: "c84eda7f94864d5d86f4fba4a485f01b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40905 } } peers { permanent_uuid: "b04e2d87b42d437f8735a536e896c858" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32933 } }
I20260620 23:40:39.571861 19468 raft_consensus.cc:2454] T 05a5fea078a741bba0813a5dbbcd8d26 P b04e2d87b42d437f8735a536e896c858 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c84eda7f94864d5d86f4fba4a485f01b in term 1.
I20260620 23:40:39.575918 19197 catalog_manager.cc:5495] T 05a5fea078a741bba0813a5dbbcd8d26 P c84eda7f94864d5d86f4fba4a485f01b reported cstate change: term changed from 0 to 1, leader changed from <none> to c84eda7f94864d5d86f4fba4a485f01b (127.0.97.1). New cstate: current_term: 1 leader_uuid: "c84eda7f94864d5d86f4fba4a485f01b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492aa9c3763c4c5d8b624a519377d16d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c84eda7f94864d5d86f4fba4a485f01b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b04e2d87b42d437f8735a536e896c858" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32933 } health_report { overall_health: UNKNOWN } } }
W20260620 23:40:39.595682 19515 tablet.cc:2369] T 05a5fea078a741bba0813a5dbbcd8d26 P b04e2d87b42d437f8735a536e896c858: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:40:39.596036 19659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:39.590088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43652 (local address 127.0.97.1:40905)
0620 23:40:39.590304 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:39.590309 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:39.590327 (+ 18us) server_negotiation.cc:407] Connection header received
0620 23:40:39.590548 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:39.590552 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:39.590633 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:39.590769 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:40:39.591214 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:39.592056 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:39.593572 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:39.594046 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:39.594175 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:39.594645 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:40:39.595326 (+ 681us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:40:39.595558 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:39.595653 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 23:40:39.595704 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 23:40:39.605340 19468 raft_consensus.cc:1261] T 05a5fea078a741bba0813a5dbbcd8d26 P b04e2d87b42d437f8735a536e896c858 [term 1 FOLLOWER]: Refusing update from remote peer c84eda7f94864d5d86f4fba4a485f01b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:40:39.605340 19599 raft_consensus.cc:1261] T 05a5fea078a741bba0813a5dbbcd8d26 P 492aa9c3763c4c5d8b624a519377d16d [term 1 FOLLOWER]: Refusing update from remote peer c84eda7f94864d5d86f4fba4a485f01b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:40:39.606221 19669 consensus_queue.cc:1035] T 05a5fea078a741bba0813a5dbbcd8d26 P c84eda7f94864d5d86f4fba4a485f01b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b04e2d87b42d437f8735a536e896c858" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:40:39.606559 19674 consensus_queue.cc:1035] T 05a5fea078a741bba0813a5dbbcd8d26 P c84eda7f94864d5d86f4fba4a485f01b [LEADER]: Connected to new peer: Peer: permanent_uuid: "492aa9c3763c4c5d8b624a519377d16d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 23:40:39.609710 19384 tablet.cc:2369] T 05a5fea078a741bba0813a5dbbcd8d26 P c84eda7f94864d5d86f4fba4a485f01b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:40:39.621567 19677 mvcc.cc:205] Tried to move back new op lower bound from 7299067247012384768 to 7299067246886023168. Current Snapshot: MvccSnapshot[applied={T|T < 7299067247012384768}]
I20260620 23:40:39.634040 19197 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:40:39.634356 19197 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:40:39.638219 19197 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f9a130e888ee412fa68f21558a38fab6: Sending DeleteTablet for 3 replicas of tablet 05a5fea078a741bba0813a5dbbcd8d26
I20260620 23:40:39.639521 19579 tablet_service.cc:1514] Processing DeleteTablet for tablet 05a5fea078a741bba0813a5dbbcd8d26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:40:39 UTC) from {username='slave'} at 127.0.0.1:60828
I20260620 23:40:39.639729 19318 tablet_service.cc:1514] Processing DeleteTablet for tablet 05a5fea078a741bba0813a5dbbcd8d26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:40:39 UTC) from {username='slave'} at 127.0.0.1:43650
I20260620 23:40:39.639778 19449 tablet_service.cc:1514] Processing DeleteTablet for tablet 05a5fea078a741bba0813a5dbbcd8d26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:40:39 UTC) from {username='slave'} at 127.0.0.1:35052
I20260620 23:40:39.641981 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 19256
I20260620 23:40:39.670318 19688 tablet_replica.cc:330] T 05a5fea078a741bba0813a5dbbcd8d26 P c84eda7f94864d5d86f4fba4a485f01b: stopping tablet replica
I20260620 23:40:39.672379 19687 tablet_replica.cc:330] T 05a5fea078a741bba0813a5dbbcd8d26 P 492aa9c3763c4c5d8b624a519377d16d: stopping tablet replica
I20260620 23:40:39.672840 19687 raft_consensus.cc:2229] T 05a5fea078a741bba0813a5dbbcd8d26 P 492aa9c3763c4c5d8b624a519377d16d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:40:39.673339 19687 raft_consensus.cc:2258] T 05a5fea078a741bba0813a5dbbcd8d26 P 492aa9c3763c4c5d8b624a519377d16d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:40:39.675292 19688 raft_consensus.cc:2229] T 05a5fea078a741bba0813a5dbbcd8d26 P c84eda7f94864d5d86f4fba4a485f01b [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:40:39.676153 19688 raft_consensus.cc:2258] T 05a5fea078a741bba0813a5dbbcd8d26 P c84eda7f94864d5d86f4fba4a485f01b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:40:39.679060 19687 ts_tablet_manager.cc:1905] T 05a5fea078a741bba0813a5dbbcd8d26 P 492aa9c3763c4c5d8b624a519377d16d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:40:39.680235 19688 ts_tablet_manager.cc:1905] T 05a5fea078a741bba0813a5dbbcd8d26 P c84eda7f94864d5d86f4fba4a485f01b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:40:39.682552 19659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:39.643784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43656 (local address 127.0.97.1:40905)
0620 23:40:39.644043 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:39.644052 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:39.669096 (+ 25044us) server_negotiation.cc:407] Connection header received
0620 23:40:39.670684 (+ 1588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:39.670694 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:39.670840 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:39.671018 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:40:39.673826 (+ 2808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:39.674697 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:39.677129 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:39.677817 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:39.677960 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:39.680787 (+ 2827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:40:39.680798 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:40:39.680817 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:40:39.681641 (+ 824us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:40:39.681707 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:40:39.681733 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:40:39.681813 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:39.682082 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 23:40:39.682201 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260620 23:40:39.683562 19689 tablet_replica.cc:330] T 05a5fea078a741bba0813a5dbbcd8d26 P b04e2d87b42d437f8735a536e896c858: stopping tablet replica
I20260620 23:40:39.684041 19378 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:40:39.684032 19689 raft_consensus.cc:2229] T 05a5fea078a741bba0813a5dbbcd8d26 P b04e2d87b42d437f8735a536e896c858 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:40:39.684587 19689 raft_consensus.cc:2258] T 05a5fea078a741bba0813a5dbbcd8d26 P b04e2d87b42d437f8735a536e896c858 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:40:39.688424 19689 ts_tablet_manager.cc:1905] T 05a5fea078a741bba0813a5dbbcd8d26 P b04e2d87b42d437f8735a536e896c858: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:40:39.688709 19687 ts_tablet_manager.cc:1918] T 05a5fea078a741bba0813a5dbbcd8d26 P 492aa9c3763c4c5d8b624a519377d16d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:40:39.688880 19687 log.cc:1196] T 05a5fea078a741bba0813a5dbbcd8d26 P 492aa9c3763c4c5d8b624a519377d16d: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998525501163-388-0/minicluster-data/ts-2/wal/wals/05a5fea078a741bba0813a5dbbcd8d26
I20260620 23:40:39.689459 19687 ts_tablet_manager.cc:1939] T 05a5fea078a741bba0813a5dbbcd8d26 P 492aa9c3763c4c5d8b624a519377d16d: Deleting consensus metadata
I20260620 23:40:39.692265 19182 catalog_manager.cc:4841] TS 492aa9c3763c4c5d8b624a519377d16d (127.0.97.3:43747): tablet 05a5fea078a741bba0813a5dbbcd8d26 (table test-table [id=1bb16e4d5d7c453994309719375133b2]) successfully deleted
I20260620 23:40:39.697425 19689 ts_tablet_manager.cc:1918] T 05a5fea078a741bba0813a5dbbcd8d26 P b04e2d87b42d437f8735a536e896c858: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:40:39.697626 19689 log.cc:1196] T 05a5fea078a741bba0813a5dbbcd8d26 P b04e2d87b42d437f8735a536e896c858: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998525501163-388-0/minicluster-data/ts-1/wal/wals/05a5fea078a741bba0813a5dbbcd8d26
I20260620 23:40:39.698252 19689 ts_tablet_manager.cc:1939] T 05a5fea078a741bba0813a5dbbcd8d26 P b04e2d87b42d437f8735a536e896c858: Deleting consensus metadata
I20260620 23:40:39.700542 19181 catalog_manager.cc:4841] TS b04e2d87b42d437f8735a536e896c858 (127.0.97.2:32933): tablet 05a5fea078a741bba0813a5dbbcd8d26 (table test-table [id=1bb16e4d5d7c453994309719375133b2]) successfully deleted
I20260620 23:40:39.798146 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 19256
W20260620 23:40:39.821717 19181 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:40905 (error 108)
I20260620 23:40:39.822531 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 19387
I20260620 23:40:39.823945 19657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:39.823329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40905 (local address 127.0.0.1:43672)
0620 23:40:39.823589 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:39.823749 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
W20260620 23:40:39.824231 19181 catalog_manager.cc:4584] TS c84eda7f94864d5d86f4fba4a485f01b (127.0.97.1:40905): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 05a5fea078a741bba0813a5dbbcd8d26: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40905: connect: Connection refused (error 111)
I20260620 23:40:39.829777 19660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:39.823356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35060 (local address 127.0.97.2:32933)
0620 23:40:39.823541 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:39.823547 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:39.823682 (+ 135us) server_negotiation.cc:407] Connection header received
0620 23:40:39.823853 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:39.823863 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:39.823976 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:39.824091 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:40:39.824611 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:39.825326 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:39.826834 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:39.827376 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:39.827474 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:39.828460 (+ 986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:40:39.828481 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:40:39.828495 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:40:39.829088 (+ 593us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:40:39.829135 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:40:39.829155 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:40:39.829242 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:39.829353 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 23:40:39.829441 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 23:40:39.830382 19509 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:40:39.878285 19657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:39.877754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40905 (local address 127.0.0.1:43678)
0620 23:40:39.878009 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:39.878111 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260620 23:40:39.879531 19657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:39.879141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40905 (local address 127.0.0.1:43688)
0620 23:40:39.879325 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:39.879397 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 23:40:39.921137 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 19387
I20260620 23:40:39.938529 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 19517
I20260620 23:40:39.940495 19657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:39.940027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40905 (local address 127.0.0.1:43696)
0620 23:40:39.940214 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:39.940292 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260620 23:40:39.941505 19657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:39.941152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40905 (local address 127.0.0.1:43704)
0620 23:40:39.941297 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:39.941374 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 23:40:39.946158 19658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:39.939330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60838 (local address 127.0.97.3:43747)
0620 23:40:39.939513 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:39.939518 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:39.939554 (+ 36us) server_negotiation.cc:407] Connection header received
0620 23:40:39.939798 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:39.939802 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:39.939903 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:39.939994 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:40:39.940450 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:39.941504 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:39.943243 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:39.943643 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:39.943740 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:39.944713 (+ 973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:40:39.944721 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:40:39.944737 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:40:39.945392 (+ 655us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:40:39.945454 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:40:39.945475 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:40:39.945558 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:39.945691 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 23:40:39.945777 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 23:40:39.946777 19639 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:40:40.022485 19657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:40.021830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40905 (local address 127.0.0.1:43718)
0620 23:40:40.022114 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:40.022237 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260620 23:40:40.023797 19657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:40.023325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40905 (local address 127.0.0.1:43734)
0620 23:40:40.023553 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:40.023660 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260620 23:40:40.037490 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 19517
I20260620 23:40:40.054271 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 19166
I20260620 23:40:40.055428 19227 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:40:40.165679 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 19166
2026-06-20T23:40:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2904 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:40:40 dist-test-slave-tcgx krb5kdc[19698](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:40:40 dist-test-slave-tcgx krb5kdc[19698](info): set up 2 sockets
Jun 20 23:40:40 dist-test-slave-tcgx krb5kdc[19698](info): commencing operation
krb5kdc: starting...
W20260620 23:40:41.291206 388 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/krb5kdc/test-user.keytab.
Jun 20 23:40:41 dist-test-slave-tcgx krb5kdc[19698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:40:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:40:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 23:40:41.550057 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42841
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42841
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:40:41.893080 19711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:40:41.893397 19711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:40:41.893577 19711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:40:41.901039 19711 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:40:41.901160 19711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:40:41.901263 19711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:40:41.901366 19711 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:40:41.901446 19711 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:40:41.911548 19711 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42841
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:40:41.912993 19711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:40:41.914392 19711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:40:41.926450 19717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:40:41.926476 19716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:40:41.926568 19719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:40:41.927178 19711 server_base.cc:1034] running on GCE node
I20260620 23:40:41.928010 19711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:40:41.929450 19711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:40:41.930687 19711 hybrid_clock.cc:648] HybridClock initialized: now 1781998841930647 us; error 54 us; skew 500 ppm
I20260620 23:40:41.931113 19711 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:40:41 dist-test-slave-tcgx krb5kdc[19698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998841, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:40:41.934306 19711 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 23:40:41.935905 19711 webserver.cc:460] Webserver started at http://127.0.97.62:35779/ using document root <none> and password file <none>
I20260620 23:40:41.936628 19711 fs_manager.cc:359] Metadata directory not provided
I20260620 23:40:41.936777 19711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:40:41.937172 19711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:40:41.940294 19711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/minicluster-data/master-0/data/instance:
uuid: "b3b5c6dd83ea4508a1fccbb36ce264f0"
format_stamp: "Formatted at 2026-06-20 23:40:41 on dist-test-slave-tcgx"
server_key: "6618b1dbcc4751a256f9665a8fb37954"
server_key_iv: "13f71964f9c4e15627a36ce9f4b052ad"
server_key_version: "encryptionkey@0"
I20260620 23:40:41.941097 19711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/minicluster-data/master-0/wal/instance:
uuid: "b3b5c6dd83ea4508a1fccbb36ce264f0"
format_stamp: "Formatted at 2026-06-20 23:40:41 on dist-test-slave-tcgx"
server_key: "6618b1dbcc4751a256f9665a8fb37954"
server_key_iv: "13f71964f9c4e15627a36ce9f4b052ad"
server_key_version: "encryptionkey@0"
I20260620 23:40:41.946470 19711 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 23:40:41.950187 19726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:41.951745 19711 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:40:41.951933 19711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/minicluster-data/master-0/wal
uuid: "b3b5c6dd83ea4508a1fccbb36ce264f0"
format_stamp: "Formatted at 2026-06-20 23:40:41 on dist-test-slave-tcgx"
server_key: "6618b1dbcc4751a256f9665a8fb37954"
server_key_iv: "13f71964f9c4e15627a36ce9f4b052ad"
server_key_version: "encryptionkey@0"
I20260620 23:40:41.952154 19711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:40:41.974397 19711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:40:41.978682 19711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:40:41.979125 19711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:40:41.998032 19711 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42841
I20260620 23:40:41.998067 19777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42841 every 8 connection(s)
I20260620 23:40:41.999512 19711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:40:42.003299 19778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:40:42.005030 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 19711
I20260620 23:40:42.005223 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/minicluster-data/master-0/wal/instance
I20260620 23:40:42.005659 388 external_mini_cluster.cc:1428] Setting key 4c329bf1e66d7b887cd34c70a599537e
I20260620 23:40:42.018172 19778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3b5c6dd83ea4508a1fccbb36ce264f0: Bootstrap starting.
Jun 20 23:40:42 dist-test-slave-tcgx krb5kdc[19698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:40:42.021744 19778 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3b5c6dd83ea4508a1fccbb36ce264f0: Neither blocks nor log segments found. Creating new log.
I20260620 23:40:42.022892 19778 log.cc:824] T 00000000000000000000000000000000 P b3b5c6dd83ea4508a1fccbb36ce264f0: Log is configured to *not* fsync() on all Append() calls
I20260620 23:40:42.026481 19778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3b5c6dd83ea4508a1fccbb36ce264f0: No bootstrap required, opened a new log
I20260620 23:40:42.031899 19781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:42.007720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58222 (local address 127.0.97.62:42841)
0620 23:40:42.008620 (+ 900us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:42.008638 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:42.008695 (+ 57us) server_negotiation.cc:407] Connection header received
0620 23:40:42.010628 (+ 1933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:42.010663 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:42.011296 (+ 633us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:42.011754 (+ 458us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:40:42.013211 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:42.015362 (+ 2151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:42.016999 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:42.018080 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:42.018279 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:42.022818 (+ 4539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:40:42.022829 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:40:42.022851 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:40:42.022912 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:40:42.027113 (+ 4201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:40:42.027827 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:40:42.027835 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:40:42.027843 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:40:42.028079 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:40:42.028459 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:40:42.028466 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:40:42.028475 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:40:42.029248 (+ 773us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:40:42.029562 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:42.029966 (+ 404us) server_negotiation.cc:299] Negotiation successful
0620 23:40:42.030436 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":347,"threads_started":1}
I20260620 23:40:42.036571 19778 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3b5c6dd83ea4508a1fccbb36ce264f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3b5c6dd83ea4508a1fccbb36ce264f0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42841 } }
I20260620 23:40:42.036922 19778 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3b5c6dd83ea4508a1fccbb36ce264f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:40:42.036998 19778 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3b5c6dd83ea4508a1fccbb36ce264f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3b5c6dd83ea4508a1fccbb36ce264f0, State: Initialized, Role: FOLLOWER
I20260620 23:40:42.037567 19778 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3b5c6dd83ea4508a1fccbb36ce264f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3b5c6dd83ea4508a1fccbb36ce264f0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42841 } }
I20260620 23:40:42.037781 19778 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3b5c6dd83ea4508a1fccbb36ce264f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:40:42.037875 19778 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3b5c6dd83ea4508a1fccbb36ce264f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:40:42.037997 19778 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3b5c6dd83ea4508a1fccbb36ce264f0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:40:42.040179 19778 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3b5c6dd83ea4508a1fccbb36ce264f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3b5c6dd83ea4508a1fccbb36ce264f0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42841 } }
I20260620 23:40:42.040813 19778 leader_election.cc:302] T 00000000000000000000000000000000 P b3b5c6dd83ea4508a1fccbb36ce264f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3b5c6dd83ea4508a1fccbb36ce264f0; no voters:
I20260620 23:40:42.041512 19778 leader_election.cc:288] T 00000000000000000000000000000000 P b3b5c6dd83ea4508a1fccbb36ce264f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:40:42.041673 19783 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3b5c6dd83ea4508a1fccbb36ce264f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:40:42.042359 19783 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3b5c6dd83ea4508a1fccbb36ce264f0 [term 1 LEADER]: Becoming Leader. State: Replica: b3b5c6dd83ea4508a1fccbb36ce264f0, State: Running, Role: LEADER
I20260620 23:40:42.042906 19783 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3b5c6dd83ea4508a1fccbb36ce264f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3b5c6dd83ea4508a1fccbb36ce264f0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42841 } }
I20260620 23:40:42.043731 19778 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3b5c6dd83ea4508a1fccbb36ce264f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:40:42.046139 19784 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3b5c6dd83ea4508a1fccbb36ce264f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3b5c6dd83ea4508a1fccbb36ce264f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3b5c6dd83ea4508a1fccbb36ce264f0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42841 } } }
I20260620 23:40:42.046181 19785 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3b5c6dd83ea4508a1fccbb36ce264f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3b5c6dd83ea4508a1fccbb36ce264f0. Latest consensus state: current_term: 1 leader_uuid: "b3b5c6dd83ea4508a1fccbb36ce264f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3b5c6dd83ea4508a1fccbb36ce264f0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42841 } } }
I20260620 23:40:42.046521 19784 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3b5c6dd83ea4508a1fccbb36ce264f0 [sys.catalog]: This master's current role is: LEADER
I20260620 23:40:42.046582 19785 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3b5c6dd83ea4508a1fccbb36ce264f0 [sys.catalog]: This master's current role is: LEADER
I20260620 23:40:42.051527 19789 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:40:42.056984 19789 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:40:42.067027 19789 catalog_manager.cc:1347] Generated new cluster ID: 4c6523661a1e4307916395e85effcf9c
I20260620 23:40:42.067163 19789 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:40:42.077378 19789 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:40:42.078706 19789 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:40:42.087047 19789 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3b5c6dd83ea4508a1fccbb36ce264f0: Generated new TSK 0
I20260620 23:40:42.087924 19789 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 23:40:42.156641 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42841
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:40:42.508371 19804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:40:42.508688 19804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:40:42.508860 19804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:40:42.516453 19804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:40:42.516579 19804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:40:42.516779 19804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 23:40:42.527369 19804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42841
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:40:42.528767 19804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:40:42.530128 19804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:40:42.544457 19809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:40:42.544562 19810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:40:42.544556 19812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:40:42.545745 19804 server_base.cc:1034] running on GCE node
I20260620 23:40:42.546362 19804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:40:42.547482 19804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:40:42.548712 19804 hybrid_clock.cc:648] HybridClock initialized: now 1781998842548656 us; error 59 us; skew 500 ppm
I20260620 23:40:42.549024 19804 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:40:42 dist-test-slave-tcgx krb5kdc[19698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998842, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:40:42.552645 19804 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 23:40:42.554316 19804 webserver.cc:460] Webserver started at http://127.0.97.1:37309/ using document root <none> and password file <none>
I20260620 23:40:42.555236 19804 fs_manager.cc:359] Metadata directory not provided
I20260620 23:40:42.555409 19804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:40:42.555765 19804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:40:42.559230 19804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/minicluster-data/ts-0/data/instance:
uuid: "f1a91d0c40264a74a94c51284fe27685"
format_stamp: "Formatted at 2026-06-20 23:40:42 on dist-test-slave-tcgx"
server_key: "0edc87b473136d14b247bf445531c8f8"
server_key_iv: "808d28b127f6deeffee5053091569096"
server_key_version: "encryptionkey@0"
I20260620 23:40:42.560122 19804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/minicluster-data/ts-0/wal/instance:
uuid: "f1a91d0c40264a74a94c51284fe27685"
format_stamp: "Formatted at 2026-06-20 23:40:42 on dist-test-slave-tcgx"
server_key: "0edc87b473136d14b247bf445531c8f8"
server_key_iv: "808d28b127f6deeffee5053091569096"
server_key_version: "encryptionkey@0"
I20260620 23:40:42.566103 19804 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260620 23:40:42.570060 19819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:42.571587 19804 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:40:42.571786 19804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/minicluster-data/ts-0/wal
uuid: "f1a91d0c40264a74a94c51284fe27685"
format_stamp: "Formatted at 2026-06-20 23:40:42 on dist-test-slave-tcgx"
server_key: "0edc87b473136d14b247bf445531c8f8"
server_key_iv: "808d28b127f6deeffee5053091569096"
server_key_version: "encryptionkey@0"
I20260620 23:40:42.572005 19804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:40:42.589812 19804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:40:42.594067 19804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:40:42.594393 19804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:40:42.595463 19804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:40:42.596845 19804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:40:42.596948 19804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:42.597103 19804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:40:42.597174 19804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:42.631184 19804 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34239
I20260620 23:40:42.631234 19931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34239 every 8 connection(s)
I20260620 23:40:42.632540 19804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
I20260620 23:40:42.636022 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 19804
I20260620 23:40:42.636224 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/minicluster-data/ts-0/wal/instance
I20260620 23:40:42.636826 388 external_mini_cluster.cc:1428] Setting key 24f6ad9e5939473e986d956e7f1be2d2
Jun 20 23:40:42 dist-test-slave-tcgx krb5kdc[19698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998842, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:40:42.656564 19934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:42.635195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54735 (local address 127.0.97.62:42841)
0620 23:40:42.635715 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:42.635722 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:42.636795 (+ 1073us) server_negotiation.cc:407] Connection header received
0620 23:40:42.639185 (+ 2390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:42.639202 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:42.639288 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:42.639483 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:40:42.641290 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:42.642455 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:42.644221 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:42.645075 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:42.645220 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:42.648918 (+ 3698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:40:42.648927 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:40:42.648935 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:40:42.648984 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:40:42.652684 (+ 3700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:40:42.653561 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:40:42.653570 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:40:42.653576 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:40:42.653732 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:40:42.654213 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:40:42.654220 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:40:42.654226 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:40:42.654622 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:40:42.654778 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:42.655765 (+ 987us) server_negotiation.cc:299] Negotiation successful
0620 23:40:42.656039 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":259,"threads_started":1}
I20260620 23:40:42.657368 19935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:42.635688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42841 (local address 127.0.97.1:54735)
0620 23:40:42.636578 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:42.636636 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 23:40:42.638834 (+ 2198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:40:42.639745 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:40:42.639755 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:40:42.640436 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:40:42.641086 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:42.641106 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:42.642637 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:42.642641 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:42.644073 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:42.644089 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:42.645295 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:42.645300 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:42.645514 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:42.646255 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:40:42.646282 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:40:42.648671 (+ 2389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:40:42.652839 (+ 4168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:40:42.652847 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:40:42.652863 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:40:42.653395 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:40:42.653868 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:40:42.653871 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:40:42.653874 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:40:42.654081 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:40:42.654779 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:40:42.654785 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:40:42.655280 (+ 495us) client_negotiation.cc:769] Sending connection context
0620 23:40:42.655569 (+ 289us) client_negotiation.cc:240] Negotiation successful
0620 23:40:42.656085 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":331,"threads_started":1}
I20260620 23:40:42.659207 19932 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42841
I20260620 23:40:42.659564 19932 heartbeater.cc:464] Registering TS with master...
I20260620 23:40:42.660537 19932 heartbeater.cc:511] Master 127.0.97.62:42841 requested a full tablet report, sending...
I20260620 23:40:42.662923 19743 ts_manager.cc:194] Registered new tserver with Master: f1a91d0c40264a74a94c51284fe27685 (127.0.97.1:34239)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 23:40:42.664573 19743 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:54735
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 23:40:42.697763 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42841
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:40:43.030853 19938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:40:43.031186 19938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:40:43.031381 19938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:40:43.038691 19938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:40:43.038820 19938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:40:43.039042 19938 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 23:40:43.049191 19938 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42841
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:40:43.050566 19938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:40:43.052016 19938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:40:43.066083 19943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:40:43.066121 19946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:40:43.066174 19944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:40:43.066711 19938 server_base.cc:1034] running on GCE node
I20260620 23:40:43.067427 19938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:40:43.068393 19938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:40:43.069594 19938 hybrid_clock.cc:648] HybridClock initialized: now 1781998843069540 us; error 46 us; skew 500 ppm
I20260620 23:40:43.069931 19938 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:40:43 dist-test-slave-tcgx krb5kdc[19698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998843, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:40:43.073109 19938 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 23:40:43.074611 19938 webserver.cc:460] Webserver started at http://127.0.97.2:42637/ using document root <none> and password file <none>
I20260620 23:40:43.075357 19938 fs_manager.cc:359] Metadata directory not provided
I20260620 23:40:43.075461 19938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:40:43.075776 19938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:40:43.078737 19938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/minicluster-data/ts-1/data/instance:
uuid: "d55718974e424a41992bcde2554c5801"
format_stamp: "Formatted at 2026-06-20 23:40:43 on dist-test-slave-tcgx"
server_key: "0057072201f5296061e4e0b7e5f74b30"
server_key_iv: "856c656e71c8a9ff480a068a86cf44bd"
server_key_version: "encryptionkey@0"
I20260620 23:40:43.079533 19938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/minicluster-data/ts-1/wal/instance:
uuid: "d55718974e424a41992bcde2554c5801"
format_stamp: "Formatted at 2026-06-20 23:40:43 on dist-test-slave-tcgx"
server_key: "0057072201f5296061e4e0b7e5f74b30"
server_key_iv: "856c656e71c8a9ff480a068a86cf44bd"
server_key_version: "encryptionkey@0"
I20260620 23:40:43.084988 19938 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 23:40:43.091480 19953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:43.093068 19938 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.003s
I20260620 23:40:43.093236 19938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/minicluster-data/ts-1/wal
uuid: "d55718974e424a41992bcde2554c5801"
format_stamp: "Formatted at 2026-06-20 23:40:43 on dist-test-slave-tcgx"
server_key: "0057072201f5296061e4e0b7e5f74b30"
server_key_iv: "856c656e71c8a9ff480a068a86cf44bd"
server_key_version: "encryptionkey@0"
I20260620 23:40:43.093436 19938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:40:43.111308 19938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:40:43.115661 19938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:40:43.115984 19938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:40:43.117024 19938 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:40:43.118454 19938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:40:43.118527 19938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:43.118702 19938 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:40:43.118762 19938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:43.153446 19938 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37419
I20260620 23:40:43.153525 20065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37419 every 8 connection(s)
I20260620 23:40:43.154778 19938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
I20260620 23:40:43.164227 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 19938
I20260620 23:40:43.164425 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/minicluster-data/ts-1/wal/instance
I20260620 23:40:43.165035 388 external_mini_cluster.cc:1428] Setting key 2a7d2d082bdf034a4bceca9dcfdd611a
Jun 20 23:40:43 dist-test-slave-tcgx krb5kdc[19698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998843, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:40:43.176789 20068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:43.157636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33811 (local address 127.0.97.62:42841)
0620 23:40:43.158160 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:43.158165 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:43.159151 (+ 986us) server_negotiation.cc:407] Connection header received
0620 23:40:43.161012 (+ 1861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:43.161018 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:43.161115 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:43.161233 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:40:43.162803 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:43.163594 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:43.165934 (+ 2340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:43.166414 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:43.166566 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:43.170792 (+ 4226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:40:43.170797 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:40:43.170800 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:40:43.170836 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:40:43.173486 (+ 2650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:40:43.174270 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:40:43.174278 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:40:43.174287 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:40:43.174424 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:40:43.174869 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:40:43.174916 (+ 47us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:40:43.174918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:40:43.175235 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:40:43.175403 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:43.176123 (+ 720us) server_negotiation.cc:299] Negotiation successful
0620 23:40:43.176376 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":322,"threads_started":1}
I20260620 23:40:43.177804 20069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:43.158233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42841 (local address 127.0.97.2:33811)
0620 23:40:43.158988 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:43.159041 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 23:40:43.160737 (+ 1696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:40:43.161488 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:40:43.161497 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:40:43.162037 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:40:43.162625 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:43.162642 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:43.163761 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:43.163766 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:43.165774 (+ 2008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:43.165787 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:43.166660 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:43.166670 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:43.166907 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:43.167768 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:40:43.167810 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:40:43.170541 (+ 2731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:40:43.173651 (+ 3110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:40:43.173660 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:40:43.173680 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:40:43.174131 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:40:43.174545 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:40:43.174550 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:40:43.174554 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:40:43.174744 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:40:43.175364 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:40:43.175375 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:40:43.175741 (+ 366us) client_negotiation.cc:769] Sending connection context
0620 23:40:43.176062 (+ 321us) client_negotiation.cc:240] Negotiation successful
0620 23:40:43.176623 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":285,"threads_started":1}
I20260620 23:40:43.179418 20066 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42841
I20260620 23:40:43.179749 20066 heartbeater.cc:464] Registering TS with master...
I20260620 23:40:43.180644 20066 heartbeater.cc:511] Master 127.0.97.62:42841 requested a full tablet report, sending...
I20260620 23:40:43.182336 19743 ts_manager.cc:194] Registered new tserver with Master: d55718974e424a41992bcde2554c5801 (127.0.97.2:37419)
I20260620 23:40:43.183650 19743 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:33811
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 23:40:43.227240 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42841
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:40:43.581951 20072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:40:43.582249 20072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:40:43.582352 20072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:40:43.589661 20072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:40:43.589792 20072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:40:43.589913 20072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 23:40:43.600399 20072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42841
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:40:43.601760 20072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:40:43.603281 20072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:40:43.619479 20077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:40:43.619503 20080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:40:43.619561 20078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:40:43.620419 20072 server_base.cc:1034] running on GCE node
I20260620 23:40:43.621198 20072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:40:43.622272 20072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:40:43.623481 20072 hybrid_clock.cc:648] HybridClock initialized: now 1781998843623441 us; error 33 us; skew 500 ppm
I20260620 23:40:43.623835 20072 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:40:43 dist-test-slave-tcgx krb5kdc[19698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998843, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:40:43.627336 20072 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 23:40:43.628892 20072 webserver.cc:460] Webserver started at http://127.0.97.3:33005/ using document root <none> and password file <none>
I20260620 23:40:43.629643 20072 fs_manager.cc:359] Metadata directory not provided
I20260620 23:40:43.629797 20072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:40:43.630123 20072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:40:43.633370 20072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/minicluster-data/ts-2/data/instance:
uuid: "d3e8712cd97e4e809ab8ef9aa62a0ca4"
format_stamp: "Formatted at 2026-06-20 23:40:43 on dist-test-slave-tcgx"
server_key: "6372fc0b972df35869931c830080ffc4"
server_key_iv: "1d56401f89f6e51e5313b856df59287f"
server_key_version: "encryptionkey@0"
I20260620 23:40:43.634171 20072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/minicluster-data/ts-2/wal/instance:
uuid: "d3e8712cd97e4e809ab8ef9aa62a0ca4"
format_stamp: "Formatted at 2026-06-20 23:40:43 on dist-test-slave-tcgx"
server_key: "6372fc0b972df35869931c830080ffc4"
server_key_iv: "1d56401f89f6e51e5313b856df59287f"
server_key_version: "encryptionkey@0"
I20260620 23:40:43.639531 20072 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 23:40:43.643407 20087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:43.645303 20072 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 23:40:43.645514 20072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/minicluster-data/ts-2/wal
uuid: "d3e8712cd97e4e809ab8ef9aa62a0ca4"
format_stamp: "Formatted at 2026-06-20 23:40:43 on dist-test-slave-tcgx"
server_key: "6372fc0b972df35869931c830080ffc4"
server_key_iv: "1d56401f89f6e51e5313b856df59287f"
server_key_version: "encryptionkey@0"
I20260620 23:40:43.645769 20072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:40:43.668000 19932 heartbeater.cc:503] Master 127.0.97.62:42841 was elected leader, sending a full tablet report...
I20260620 23:40:43.670528 20072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:40:43.675642 20072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:40:43.675951 20072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:40:43.677017 20072 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:40:43.678467 20072 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:40:43.678543 20072 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:43.678658 20072 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:40:43.678725 20072 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:43.712654 20072 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36257
I20260620 23:40:43.712706 20199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36257 every 8 connection(s)
I20260620 23:40:43.714193 20072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
I20260620 23:40:43.724375 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 20072
I20260620 23:40:43.724574 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/minicluster-data/ts-2/wal/instance
I20260620 23:40:43.725071 388 external_mini_cluster.cc:1428] Setting key 4958d621bd07d97243b936a92aaad5ee
Jun 20 23:40:43 dist-test-slave-tcgx krb5kdc[19698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998843, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:40:43.735773 20202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:43.716958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42451 (local address 127.0.97.62:42841)
0620 23:40:43.717402 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:43.717407 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:43.718470 (+ 1063us) server_negotiation.cc:407] Connection header received
0620 23:40:43.720412 (+ 1942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:43.720416 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:43.720497 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:43.720605 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:40:43.722265 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:43.723095 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:43.724938 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:43.725548 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:43.725648 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:43.729850 (+ 4202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:40:43.729855 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:40:43.729859 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:40:43.729895 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:40:43.732586 (+ 2691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:40:43.733362 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:40:43.733366 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:40:43.733368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:40:43.733512 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:40:43.733946 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:40:43.733949 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:40:43.733951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:40:43.734229 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:40:43.734358 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:43.735288 (+ 930us) server_negotiation.cc:299] Negotiation successful
0620 23:40:43.735469 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":200,"threads_started":1}
I20260620 23:40:43.736974 20203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:43.717569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42841 (local address 127.0.97.3:42451)
0620 23:40:43.718293 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:43.718330 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 23:40:43.720149 (+ 1819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:40:43.720889 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:40:43.720898 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:40:43.721458 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:40:43.722100 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:43.722118 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:43.723235 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:43.723239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:43.724785 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:43.724798 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:43.725924 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:43.725934 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:43.726166 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:43.727056 (+ 890us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:40:43.727114 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:40:43.729632 (+ 2518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:40:43.732753 (+ 3121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:40:43.732761 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:40:43.732776 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:40:43.733253 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:40:43.733627 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:40:43.733630 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:40:43.733633 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:40:43.733816 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:40:43.734400 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:40:43.734409 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:40:43.734836 (+ 427us) client_negotiation.cc:769] Sending connection context
0620 23:40:43.735206 (+ 370us) client_negotiation.cc:240] Negotiation successful
0620 23:40:43.735628 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":316,"threads_started":1}
I20260620 23:40:43.738500 20200 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42841
I20260620 23:40:43.738909 20200 heartbeater.cc:464] Registering TS with master...
I20260620 23:40:43.739782 20200 heartbeater.cc:511] Master 127.0.97.62:42841 requested a full tablet report, sending...
I20260620 23:40:43.741295 19743 ts_manager.cc:194] Registered new tserver with Master: d3e8712cd97e4e809ab8ef9aa62a0ca4 (127.0.97.3:36257)
I20260620 23:40:43.742069 19743 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:42451
I20260620 23:40:43.753335 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:40:43.770538 20202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:43.757857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58230 (local address 127.0.97.62:42841)
0620 23:40:43.758069 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:43.758074 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:43.758565 (+ 491us) server_negotiation.cc:407] Connection header received
0620 23:40:43.758753 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:43.758758 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:43.758831 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:43.759000 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:40:43.760343 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:43.761121 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:43.762539 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:43.763013 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:43.763114 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:43.765371 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:40:43.765376 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:40:43.765379 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:40:43.765413 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:40:43.768150 (+ 2737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:40:43.768799 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:40:43.768804 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:40:43.768806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:40:43.768949 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:40:43.769278 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:40:43.769281 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:40:43.769283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:40:43.769630 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:40:43.769811 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:43.770107 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 23:40:43.770291 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 23:40:43.774423 19743 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58230:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:40:43.781957 19743 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:40:43.804533 20214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:43.795248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46550 (local address 127.0.97.2:37419)
0620 23:40:43.795787 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:43.795799 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:43.795828 (+ 29us) server_negotiation.cc:407] Connection header received
0620 23:40:43.795929 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:43.795937 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:43.796205 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:43.796406 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:40:43.797394 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:43.798983 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:43.801876 (+ 2893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:43.803309 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:43.803433 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:43.803631 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:43.804066 (+ 435us) server_negotiation.cc:299] Negotiation successful
0620 23:40:43.804238 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":239,"threads_started":1}
I20260620 23:40:43.804621 20213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:43.794668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37419 (local address 127.0.0.1:46550)
0620 23:40:43.795484 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:43.795542 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 23:40:43.795709 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:40:43.796546 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:40:43.796556 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:40:43.796593 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:40:43.797079 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:43.797102 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:43.799158 (+ 2056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:43.799168 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:43.801677 (+ 2509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:43.801692 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:43.803516 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:43.803523 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:43.803784 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:43.803842 (+ 58us) client_negotiation.cc:769] Sending connection context
0620 23:40:43.803996 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 23:40:43.804180 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"mutex_wait_us":22,"thread_start_us":403,"threads_started":1}
I20260620 23:40:43.805579 20215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:43.795941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36257 (local address 127.0.0.1:46154)
0620 23:40:43.796557 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:43.796577 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:40:43.796701 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:40:43.798052 (+ 1351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:40:43.798060 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:40:43.798093 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:40:43.798585 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:43.798597 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:43.800904 (+ 2307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:43.800912 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:43.802962 (+ 2050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:43.802975 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:43.804748 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:43.804762 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:43.804933 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:43.804969 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 23:40:43.805038 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 23:40:43.805175 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"mutex_wait_us":4,"thread_start_us":657,"threads_started":1}
I20260620 23:40:43.807381 20218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:43.796902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44460 (local address 127.0.97.1:34239)
0620 23:40:43.798844 (+ 1942us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:43.798852 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:43.799469 (+ 617us) server_negotiation.cc:407] Connection header received
0620 23:40:43.799658 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:43.799666 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:43.799918 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:43.800111 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:40:43.800950 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:43.802530 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:43.805037 (+ 2507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:43.806336 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:43.806493 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:43.806701 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:43.806911 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 23:40:43.807101 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1786,"thread_start_us":1634,"threads_started":1}
I20260620 23:40:43.807395 20216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:43.796133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34239 (local address 127.0.0.1:44460)
0620 23:40:43.799331 (+ 3198us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:43.799355 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 23:40:43.799490 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:40:43.800205 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:40:43.800212 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:40:43.800281 (+ 69us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:40:43.800749 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:43.800760 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:43.802677 (+ 1917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:43.802683 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:43.804892 (+ 2209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:43.804903 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:43.806573 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:43.806579 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:43.806742 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:43.806772 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 23:40:43.806869 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 23:40:43.807020 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3054,"mutex_wait_us":1,"thread_start_us":2875,"threads_started":1}
I20260620 23:40:43.809092 20001 tablet_service.cc:1467] Processing CreateTablet for tablet d0987491be054a558acf0f58562ca44c (DEFAULT_TABLE table=test-table [id=e376444c8c87463dae931ced5395f73f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:40:43.810676 20001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0987491be054a558acf0f58562ca44c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:40:43.811285 19867 tablet_service.cc:1467] Processing CreateTablet for tablet d0987491be054a558acf0f58562ca44c (DEFAULT_TABLE table=test-table [id=e376444c8c87463dae931ced5395f73f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:40:43.812677 19867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0987491be054a558acf0f58562ca44c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:40:43.815268 20217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:43.796471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46154 (local address 127.0.97.3:36257)
0620 23:40:43.797291 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:43.797302 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:43.797340 (+ 38us) server_negotiation.cc:407] Connection header received
0620 23:40:43.797466 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:43.797476 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:43.797816 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:43.798815 (+ 999us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:40:43.799198 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:43.800767 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:43.803231 (+ 2464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:43.804480 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:43.805431 (+ 951us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:43.805874 (+ 443us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:43.805959 (+ 85us) server_negotiation.cc:299] Negotiation successful
0620 23:40:43.806130 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":493,"threads_started":1}
I20260620 23:40:43.818707 20135 tablet_service.cc:1467] Processing CreateTablet for tablet d0987491be054a558acf0f58562ca44c (DEFAULT_TABLE table=test-table [id=e376444c8c87463dae931ced5395f73f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:40:43.819965 20135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0987491be054a558acf0f58562ca44c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:40:43.826236 20219 tablet_bootstrap.cc:492] T d0987491be054a558acf0f58562ca44c P d55718974e424a41992bcde2554c5801: Bootstrap starting.
I20260620 23:40:43.828574 20220 tablet_bootstrap.cc:492] T d0987491be054a558acf0f58562ca44c P f1a91d0c40264a74a94c51284fe27685: Bootstrap starting.
I20260620 23:40:43.830320 20219 tablet_bootstrap.cc:654] T d0987491be054a558acf0f58562ca44c P d55718974e424a41992bcde2554c5801: Neither blocks nor log segments found. Creating new log.
I20260620 23:40:43.831728 20220 tablet_bootstrap.cc:654] T d0987491be054a558acf0f58562ca44c P f1a91d0c40264a74a94c51284fe27685: Neither blocks nor log segments found. Creating new log.
I20260620 23:40:43.831753 20219 log.cc:824] T d0987491be054a558acf0f58562ca44c P d55718974e424a41992bcde2554c5801: Log is configured to *not* fsync() on all Append() calls
I20260620 23:40:43.832963 20220 log.cc:824] T d0987491be054a558acf0f58562ca44c P f1a91d0c40264a74a94c51284fe27685: Log is configured to *not* fsync() on all Append() calls
I20260620 23:40:43.835585 20222 tablet_bootstrap.cc:492] T d0987491be054a558acf0f58562ca44c P d3e8712cd97e4e809ab8ef9aa62a0ca4: Bootstrap starting.
I20260620 23:40:43.835719 20219 tablet_bootstrap.cc:492] T d0987491be054a558acf0f58562ca44c P d55718974e424a41992bcde2554c5801: No bootstrap required, opened a new log
I20260620 23:40:43.836066 20219 ts_tablet_manager.cc:1397] T d0987491be054a558acf0f58562ca44c P d55718974e424a41992bcde2554c5801: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.002s
I20260620 23:40:43.839030 20222 tablet_bootstrap.cc:654] T d0987491be054a558acf0f58562ca44c P d3e8712cd97e4e809ab8ef9aa62a0ca4: Neither blocks nor log segments found. Creating new log.
I20260620 23:40:43.839448 20220 tablet_bootstrap.cc:492] T d0987491be054a558acf0f58562ca44c P f1a91d0c40264a74a94c51284fe27685: No bootstrap required, opened a new log
I20260620 23:40:43.839740 20220 ts_tablet_manager.cc:1397] T d0987491be054a558acf0f58562ca44c P f1a91d0c40264a74a94c51284fe27685: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20260620 23:40:43.840196 20222 log.cc:824] T d0987491be054a558acf0f58562ca44c P d3e8712cd97e4e809ab8ef9aa62a0ca4: Log is configured to *not* fsync() on all Append() calls
I20260620 23:40:43.842928 20222 tablet_bootstrap.cc:492] T d0987491be054a558acf0f58562ca44c P d3e8712cd97e4e809ab8ef9aa62a0ca4: No bootstrap required, opened a new log
I20260620 23:40:43.843256 20222 ts_tablet_manager.cc:1397] T d0987491be054a558acf0f58562ca44c P d3e8712cd97e4e809ab8ef9aa62a0ca4: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.003s
I20260620 23:40:43.847036 20219 raft_consensus.cc:348] T d0987491be054a558acf0f58562ca44c P d55718974e424a41992bcde2554c5801 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d55718974e424a41992bcde2554c5801" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37419 } } peers { permanent_uuid: "d3e8712cd97e4e809ab8ef9aa62a0ca4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36257 } } peers { permanent_uuid: "f1a91d0c40264a74a94c51284fe27685" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34239 } }
I20260620 23:40:43.847541 20219 raft_consensus.cc:374] T d0987491be054a558acf0f58562ca44c P d55718974e424a41992bcde2554c5801 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:40:43.847635 20219 raft_consensus.cc:729] T d0987491be054a558acf0f58562ca44c P d55718974e424a41992bcde2554c5801 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d55718974e424a41992bcde2554c5801, State: Initialized, Role: FOLLOWER
I20260620 23:40:43.848312 20219 consensus_queue.cc:260] T d0987491be054a558acf0f58562ca44c P d55718974e424a41992bcde2554c5801 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d55718974e424a41992bcde2554c5801" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37419 } } peers { permanent_uuid: "d3e8712cd97e4e809ab8ef9aa62a0ca4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36257 } } peers { permanent_uuid: "f1a91d0c40264a74a94c51284fe27685" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34239 } }
I20260620 23:40:43.849556 20066 heartbeater.cc:503] Master 127.0.97.62:42841 was elected leader, sending a full tablet report...
I20260620 23:40:43.850347 20219 ts_tablet_manager.cc:1428] T d0987491be054a558acf0f58562ca44c P d55718974e424a41992bcde2554c5801: Time spent starting tablet: real 0.014s user 0.009s sys 0.005s
I20260620 23:40:43.850416 20220 raft_consensus.cc:348] T d0987491be054a558acf0f58562ca44c P f1a91d0c40264a74a94c51284fe27685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d55718974e424a41992bcde2554c5801" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37419 } } peers { permanent_uuid: "d3e8712cd97e4e809ab8ef9aa62a0ca4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36257 } } peers { permanent_uuid: "f1a91d0c40264a74a94c51284fe27685" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34239 } }
I20260620 23:40:43.850854 20220 raft_consensus.cc:374] T d0987491be054a558acf0f58562ca44c P f1a91d0c40264a74a94c51284fe27685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:40:43.850998 20220 raft_consensus.cc:729] T d0987491be054a558acf0f58562ca44c P f1a91d0c40264a74a94c51284fe27685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a91d0c40264a74a94c51284fe27685, State: Initialized, Role: FOLLOWER
I20260620 23:40:43.851692 20220 consensus_queue.cc:260] T d0987491be054a558acf0f58562ca44c P f1a91d0c40264a74a94c51284fe27685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d55718974e424a41992bcde2554c5801" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37419 } } peers { permanent_uuid: "d3e8712cd97e4e809ab8ef9aa62a0ca4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36257 } } peers { permanent_uuid: "f1a91d0c40264a74a94c51284fe27685" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34239 } }
I20260620 23:40:43.852758 20222 raft_consensus.cc:348] T d0987491be054a558acf0f58562ca44c P d3e8712cd97e4e809ab8ef9aa62a0ca4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d55718974e424a41992bcde2554c5801" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37419 } } peers { permanent_uuid: "d3e8712cd97e4e809ab8ef9aa62a0ca4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36257 } } peers { permanent_uuid: "f1a91d0c40264a74a94c51284fe27685" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34239 } }
I20260620 23:40:43.853147 20222 raft_consensus.cc:374] T d0987491be054a558acf0f58562ca44c P d3e8712cd97e4e809ab8ef9aa62a0ca4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:40:43.853266 20222 raft_consensus.cc:729] T d0987491be054a558acf0f58562ca44c P d3e8712cd97e4e809ab8ef9aa62a0ca4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3e8712cd97e4e809ab8ef9aa62a0ca4, State: Initialized, Role: FOLLOWER
I20260620 23:40:43.853856 20222 consensus_queue.cc:260] T d0987491be054a558acf0f58562ca44c P d3e8712cd97e4e809ab8ef9aa62a0ca4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d55718974e424a41992bcde2554c5801" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37419 } } peers { permanent_uuid: "d3e8712cd97e4e809ab8ef9aa62a0ca4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36257 } } peers { permanent_uuid: "f1a91d0c40264a74a94c51284fe27685" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34239 } }
I20260620 23:40:43.854724 20220 ts_tablet_manager.cc:1428] T d0987491be054a558acf0f58562ca44c P f1a91d0c40264a74a94c51284fe27685: Time spent starting tablet: real 0.015s user 0.006s sys 0.008s
I20260620 23:40:43.855829 20200 heartbeater.cc:503] Master 127.0.97.62:42841 was elected leader, sending a full tablet report...
I20260620 23:40:43.856457 20222 ts_tablet_manager.cc:1428] T d0987491be054a558acf0f58562ca44c P d3e8712cd97e4e809ab8ef9aa62a0ca4: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260620 23:40:43.867417 20225 raft_consensus.cc:482] T d0987491be054a558acf0f58562ca44c P d55718974e424a41992bcde2554c5801 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:40:43.867738 20225 raft_consensus.cc:504] T d0987491be054a558acf0f58562ca44c P d55718974e424a41992bcde2554c5801 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d55718974e424a41992bcde2554c5801" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37419 } } peers { permanent_uuid: "d3e8712cd97e4e809ab8ef9aa62a0ca4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36257 } } peers { permanent_uuid: "f1a91d0c40264a74a94c51284fe27685" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34239 } }
I20260620 23:40:43.869390 20225 leader_election.cc:288] T d0987491be054a558acf0f58562ca44c P d55718974e424a41992bcde2554c5801 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3e8712cd97e4e809ab8ef9aa62a0ca4 (127.0.97.3:36257), f1a91d0c40264a74a94c51284fe27685 (127.0.97.1:34239)
I20260620 23:40:43.877460 20217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:43.869849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59391 (local address 127.0.97.3:36257)
0620 23:40:43.870081 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:43.870091 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:43.870896 (+ 805us) server_negotiation.cc:407] Connection header received
0620 23:40:43.871115 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:43.871119 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:43.871207 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:43.871299 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:40:43.871925 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:43.872710 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:43.874831 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:43.876125 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:43.876264 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:43.876370 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:43.877057 (+ 687us) server_negotiation.cc:299] Negotiation successful
0620 23:40:43.877190 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 23:40:43.877460 20218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:43.870070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35039 (local address 127.0.97.1:34239)
0620 23:40:43.870285 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:43.870292 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:43.870759 (+ 467us) server_negotiation.cc:407] Connection header received
0620 23:40:43.871014 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:43.871023 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:43.871115 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:43.871185 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:40:43.871816 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:43.872537 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:43.874827 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:43.876133 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:43.876264 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:43.876378 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:43.877057 (+ 679us) server_negotiation.cc:299] Negotiation successful
0620 23:40:43.877190 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 23:40:43.877676 20228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:43.869652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34239 (local address 127.0.97.2:35039)
0620 23:40:43.870624 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:43.870648 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 23:40:43.870805 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:40:43.871225 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:40:43.871232 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:40:43.871261 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:40:43.871668 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:43.871680 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:43.872664 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:43.872671 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:43.874636 (+ 1965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:43.874654 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:43.876513 (+ 1859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:43.876518 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:43.876802 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:43.876914 (+ 112us) client_negotiation.cc:769] Sending connection context
0620 23:40:43.877016 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 23:40:43.877195 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":791,"mutex_wait_us":202,"thread_start_us":550,"threads_started":1}
I20260620 23:40:43.878118 20229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:43.869750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36257 (local address 127.0.97.2:59391)
0620 23:40:43.870788 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:43.870806 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:40:43.870978 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:40:43.871334 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:40:43.871339 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:40:43.871382 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:40:43.871719 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:43.871726 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:43.872844 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:43.872847 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:43.874698 (+ 1851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:43.874712 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:43.876504 (+ 1792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:43.876512 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:43.876806 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:43.876914 (+ 108us) client_negotiation.cc:769] Sending connection context
0620 23:40:43.877016 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 23:40:43.877190 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":898,"mutex_wait_us":117,"thread_start_us":731,"threads_started":1}
I20260620 23:40:43.878733 19887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0987491be054a558acf0f58562ca44c" candidate_uuid: "d55718974e424a41992bcde2554c5801" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a91d0c40264a74a94c51284fe27685" is_pre_election: true
I20260620 23:40:43.879171 20155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0987491be054a558acf0f58562ca44c" candidate_uuid: "d55718974e424a41992bcde2554c5801" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3e8712cd97e4e809ab8ef9aa62a0ca4" is_pre_election: true
I20260620 23:40:43.879354 19887 raft_consensus.cc:2454] T d0987491be054a558acf0f58562ca44c P f1a91d0c40264a74a94c51284fe27685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d55718974e424a41992bcde2554c5801 in term 0.
I20260620 23:40:43.879601 20155 raft_consensus.cc:2454] T d0987491be054a558acf0f58562ca44c P d3e8712cd97e4e809ab8ef9aa62a0ca4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d55718974e424a41992bcde2554c5801 in term 0.
I20260620 23:40:43.880101 19957 leader_election.cc:302] T d0987491be054a558acf0f58562ca44c P d55718974e424a41992bcde2554c5801 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d55718974e424a41992bcde2554c5801, f1a91d0c40264a74a94c51284fe27685; no voters:
I20260620 23:40:43.880463 20225 raft_consensus.cc:2781] T d0987491be054a558acf0f58562ca44c P d55718974e424a41992bcde2554c5801 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:40:43.880616 20225 raft_consensus.cc:482] T d0987491be054a558acf0f58562ca44c P d55718974e424a41992bcde2554c5801 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:40:43.880720 20225 raft_consensus.cc:3037] T d0987491be054a558acf0f58562ca44c P d55718974e424a41992bcde2554c5801 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:40:43.883015 20225 raft_consensus.cc:504] T d0987491be054a558acf0f58562ca44c P d55718974e424a41992bcde2554c5801 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d55718974e424a41992bcde2554c5801" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37419 } } peers { permanent_uuid: "d3e8712cd97e4e809ab8ef9aa62a0ca4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36257 } } peers { permanent_uuid: "f1a91d0c40264a74a94c51284fe27685" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34239 } }
I20260620 23:40:43.883662 20225 leader_election.cc:288] T d0987491be054a558acf0f58562ca44c P d55718974e424a41992bcde2554c5801 [CANDIDATE]: Term 1 election: Requested vote from peers d3e8712cd97e4e809ab8ef9aa62a0ca4 (127.0.97.3:36257), f1a91d0c40264a74a94c51284fe27685 (127.0.97.1:34239)
I20260620 23:40:43.884087 20155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0987491be054a558acf0f58562ca44c" candidate_uuid: "d55718974e424a41992bcde2554c5801" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3e8712cd97e4e809ab8ef9aa62a0ca4"
I20260620 23:40:43.884155 19887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0987491be054a558acf0f58562ca44c" candidate_uuid: "d55718974e424a41992bcde2554c5801" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a91d0c40264a74a94c51284fe27685"
I20260620 23:40:43.884297 20155 raft_consensus.cc:3037] T d0987491be054a558acf0f58562ca44c P d3e8712cd97e4e809ab8ef9aa62a0ca4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:40:43.884413 19887 raft_consensus.cc:3037] T d0987491be054a558acf0f58562ca44c P f1a91d0c40264a74a94c51284fe27685 [term 0 FOLLOWER]: Advancing to term 1
W20260620 23:40:43.885020 19933 tablet.cc:2369] T d0987491be054a558acf0f58562ca44c P f1a91d0c40264a74a94c51284fe27685: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:40:43.886687 20155 raft_consensus.cc:2454] T d0987491be054a558acf0f58562ca44c P d3e8712cd97e4e809ab8ef9aa62a0ca4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d55718974e424a41992bcde2554c5801 in term 1.
I20260620 23:40:43.887238 19956 leader_election.cc:302] T d0987491be054a558acf0f58562ca44c P d55718974e424a41992bcde2554c5801 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3e8712cd97e4e809ab8ef9aa62a0ca4, d55718974e424a41992bcde2554c5801; no voters:
I20260620 23:40:43.887418 19887 raft_consensus.cc:2454] T d0987491be054a558acf0f58562ca44c P f1a91d0c40264a74a94c51284fe27685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d55718974e424a41992bcde2554c5801 in term 1.
I20260620 23:40:43.887643 20225 raft_consensus.cc:2781] T d0987491be054a558acf0f58562ca44c P d55718974e424a41992bcde2554c5801 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:40:43.888242 20225 raft_consensus.cc:686] T d0987491be054a558acf0f58562ca44c P d55718974e424a41992bcde2554c5801 [term 1 LEADER]: Becoming Leader. State: Replica: d55718974e424a41992bcde2554c5801, State: Running, Role: LEADER
I20260620 23:40:43.888723 20225 consensus_queue.cc:237] T d0987491be054a558acf0f58562ca44c P d55718974e424a41992bcde2554c5801 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d55718974e424a41992bcde2554c5801" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37419 } } peers { permanent_uuid: "d3e8712cd97e4e809ab8ef9aa62a0ca4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36257 } } peers { permanent_uuid: "f1a91d0c40264a74a94c51284fe27685" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34239 } }
I20260620 23:40:43.893708 19743 catalog_manager.cc:5495] T d0987491be054a558acf0f58562ca44c P d55718974e424a41992bcde2554c5801 reported cstate change: term changed from 0 to 1, leader changed from <none> to d55718974e424a41992bcde2554c5801 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "d55718974e424a41992bcde2554c5801" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d55718974e424a41992bcde2554c5801" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37419 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3e8712cd97e4e809ab8ef9aa62a0ca4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1a91d0c40264a74a94c51284fe27685" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34239 } health_report { overall_health: UNKNOWN } } }
W20260620 23:40:43.907305 20067 tablet.cc:2369] T d0987491be054a558acf0f58562ca44c P d55718974e424a41992bcde2554c5801: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:40:43.930338 20214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:43.925337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46556 (local address 127.0.97.2:37419)
0620 23:40:43.925514 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:43.925519 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:43.925625 (+ 106us) server_negotiation.cc:407] Connection header received
0620 23:40:43.925800 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:43.925804 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:43.925879 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:43.926043 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:40:43.926345 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:43.927124 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:43.928481 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:43.928873 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:43.928963 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:43.929408 (+ 445us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:40:43.929894 (+ 486us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:40:43.930022 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:43.930062 (+ 40us) server_negotiation.cc:299] Negotiation successful
0620 23:40:43.930123 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 23:40:43.938644 19887 raft_consensus.cc:1261] T d0987491be054a558acf0f58562ca44c P f1a91d0c40264a74a94c51284fe27685 [term 1 FOLLOWER]: Refusing update from remote peer d55718974e424a41992bcde2554c5801: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:40:43.938640 20155 raft_consensus.cc:1261] T d0987491be054a558acf0f58562ca44c P d3e8712cd97e4e809ab8ef9aa62a0ca4 [term 1 FOLLOWER]: Refusing update from remote peer d55718974e424a41992bcde2554c5801: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:40:43.939762 20230 consensus_queue.cc:1035] T d0987491be054a558acf0f58562ca44c P d55718974e424a41992bcde2554c5801 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a91d0c40264a74a94c51284fe27685" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:40:43.940104 20225 consensus_queue.cc:1035] T d0987491be054a558acf0f58562ca44c P d55718974e424a41992bcde2554c5801 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3e8712cd97e4e809ab8ef9aa62a0ca4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:40:43.968844 19743 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58230:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:40:43.969133 19743 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58230:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:40:43.972844 19743 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b3b5c6dd83ea4508a1fccbb36ce264f0: Sending DeleteTablet for 3 replicas of tablet d0987491be054a558acf0f58562ca44c
I20260620 23:40:43.974217 19867 tablet_service.cc:1514] Processing DeleteTablet for tablet d0987491be054a558acf0f58562ca44c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:40:43 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44460
I20260620 23:40:43.974222 20001 tablet_service.cc:1514] Processing DeleteTablet for tablet d0987491be054a558acf0f58562ca44c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:40:43 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46550
I20260620 23:40:43.974582 20135 tablet_service.cc:1514] Processing DeleteTablet for tablet d0987491be054a558acf0f58562ca44c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:40:43 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46154
I20260620 23:40:43.975899 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 19804
I20260620 23:40:43.993656 20243 tablet_replica.cc:330] T d0987491be054a558acf0f58562ca44c P d55718974e424a41992bcde2554c5801: stopping tablet replica
I20260620 23:40:43.994338 20243 raft_consensus.cc:2229] T d0987491be054a558acf0f58562ca44c P d55718974e424a41992bcde2554c5801 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:40:43.995354 20243 raft_consensus.cc:2258] T d0987491be054a558acf0f58562ca44c P d55718974e424a41992bcde2554c5801 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:40:43.996173 20242 tablet_replica.cc:330] T d0987491be054a558acf0f58562ca44c P f1a91d0c40264a74a94c51284fe27685: stopping tablet replica
I20260620 23:40:43.996781 20242 raft_consensus.cc:2229] T d0987491be054a558acf0f58562ca44c P f1a91d0c40264a74a94c51284fe27685 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:40:43.997298 20242 raft_consensus.cc:2258] T d0987491be054a558acf0f58562ca44c P f1a91d0c40264a74a94c51284fe27685 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:40:44.000236 20243 ts_tablet_manager.cc:1905] T d0987491be054a558acf0f58562ca44c P d55718974e424a41992bcde2554c5801: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:40:44.000260 20242 ts_tablet_manager.cc:1905] T d0987491be054a558acf0f58562ca44c P f1a91d0c40264a74a94c51284fe27685: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:40:43.998075 20244 tablet_replica.cc:330] T d0987491be054a558acf0f58562ca44c P d3e8712cd97e4e809ab8ef9aa62a0ca4: stopping tablet replica
I20260620 23:40:44.000939 20244 raft_consensus.cc:2229] T d0987491be054a558acf0f58562ca44c P d3e8712cd97e4e809ab8ef9aa62a0ca4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:40:44.001515 20244 raft_consensus.cc:2258] T d0987491be054a558acf0f58562ca44c P d3e8712cd97e4e809ab8ef9aa62a0ca4 [term 1 FOLLOWER]: Raft consensus is shut down!
Jun 20 23:40:43 dist-test-slave-tcgx krb5kdc[19698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 23:40:44.006376 20244 ts_tablet_manager.cc:1905] T d0987491be054a558acf0f58562ca44c P d3e8712cd97e4e809ab8ef9aa62a0ca4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:40:44.007609 20242 ts_tablet_manager.cc:1918] T d0987491be054a558acf0f58562ca44c P f1a91d0c40264a74a94c51284fe27685: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:40:44.007809 20242 log.cc:1196] T d0987491be054a558acf0f58562ca44c P f1a91d0c40264a74a94c51284fe27685: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/minicluster-data/ts-0/wal/wals/d0987491be054a558acf0f58562ca44c
I20260620 23:40:44.008471 20242 ts_tablet_manager.cc:1939] T d0987491be054a558acf0f58562ca44c P f1a91d0c40264a74a94c51284fe27685: Deleting consensus metadata
I20260620 23:40:44.011256 19730 catalog_manager.cc:4841] TS f1a91d0c40264a74a94c51284fe27685 (127.0.97.1:34239): tablet d0987491be054a558acf0f58562ca44c (table test-table [id=e376444c8c87463dae931ced5395f73f]) successfully deleted
I20260620 23:40:44.015174 20218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:43.978228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44464 (local address 127.0.97.1:34239)
0620 23:40:43.978435 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:43.978442 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:43.983087 (+ 4645us) server_negotiation.cc:407] Connection header received
0620 23:40:43.984115 (+ 1028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:43.984124 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:43.984242 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:43.984337 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:40:43.988715 (+ 4378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:43.990291 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:43.995520 (+ 5229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:43.996284 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:43.996388 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:44.006761 (+ 10373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:40:44.006779 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:40:44.006798 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:40:44.006853 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:40:44.010876 (+ 4023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:40:44.012398 (+ 1522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:40:44.012409 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:40:44.012412 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:40:44.012585 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:40:44.013112 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:40:44.013119 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:40:44.013123 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:40:44.013577 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:40:44.013863 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:44.014444 (+ 581us) server_negotiation.cc:299] Negotiation successful
0620 23:40:44.014696 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 23:40:44.016698 19927 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:40:44.017961 20243 ts_tablet_manager.cc:1918] T d0987491be054a558acf0f58562ca44c P d55718974e424a41992bcde2554c5801: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:40:44.018136 20243 log.cc:1196] T d0987491be054a558acf0f58562ca44c P d55718974e424a41992bcde2554c5801: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/minicluster-data/ts-1/wal/wals/d0987491be054a558acf0f58562ca44c
I20260620 23:40:44.018666 20243 ts_tablet_manager.cc:1939] T d0987491be054a558acf0f58562ca44c P d55718974e424a41992bcde2554c5801: Deleting consensus metadata
I20260620 23:40:44.020648 19729 catalog_manager.cc:4841] TS d55718974e424a41992bcde2554c5801 (127.0.97.2:37419): tablet d0987491be054a558acf0f58562ca44c (table test-table [id=e376444c8c87463dae931ced5395f73f]) successfully deleted
I20260620 23:40:44.022070 20244 ts_tablet_manager.cc:1918] T d0987491be054a558acf0f58562ca44c P d3e8712cd97e4e809ab8ef9aa62a0ca4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:40:44.022368 20244 log.cc:1196] T d0987491be054a558acf0f58562ca44c P d3e8712cd97e4e809ab8ef9aa62a0ca4: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998525501163-388-0/minicluster-data/ts-2/wal/wals/d0987491be054a558acf0f58562ca44c
I20260620 23:40:44.023135 20244 ts_tablet_manager.cc:1939] T d0987491be054a558acf0f58562ca44c P d3e8712cd97e4e809ab8ef9aa62a0ca4: Deleting consensus metadata
I20260620 23:40:44.025569 19729 catalog_manager.cc:4841] TS d3e8712cd97e4e809ab8ef9aa62a0ca4 (127.0.97.3:36257): tablet d0987491be054a558acf0f58562ca44c (table test-table [id=e376444c8c87463dae931ced5395f73f]) successfully deleted
I20260620 23:40:44.128158 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 19804
I20260620 23:40:44.147445 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 19938
Jun 20 23:40:44 dist-test-slave-tcgx krb5kdc[19698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 23:40:44.166731 20214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:44.148277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46562 (local address 127.0.97.2:37419)
0620 23:40:44.148471 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:44.148478 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:44.148499 (+ 21us) server_negotiation.cc:407] Connection header received
0620 23:40:44.148711 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:44.148717 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:44.148796 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:44.148912 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:40:44.150207 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:44.151129 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:44.152550 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:44.153218 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:44.153367 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:44.158980 (+ 5613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:40:44.158995 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:40:44.159014 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:40:44.159060 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:40:44.163383 (+ 4323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:40:44.164289 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:40:44.164299 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:40:44.164303 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:40:44.164489 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:40:44.164922 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:40:44.164928 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:40:44.164932 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:40:44.165443 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:40:44.165766 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:44.166067 (+ 301us) server_negotiation.cc:299] Negotiation successful
0620 23:40:44.166313 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 23:40:44.167681 20061 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:40:44.284338 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 19938
I20260620 23:40:44.303665 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 20072
Jun 20 23:40:44 dist-test-slave-tcgx krb5kdc[19698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 23:40:44.319157 20217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:44.304460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46168 (local address 127.0.97.3:36257)
0620 23:40:44.304636 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:44.304641 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:44.304660 (+ 19us) server_negotiation.cc:407] Connection header received
0620 23:40:44.304793 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:44.304797 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:44.304936 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:44.305025 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:40:44.306476 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:44.307357 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:44.308759 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:44.309181 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:44.309281 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:44.313694 (+ 4413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:40:44.313710 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:40:44.313727 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:40:44.313767 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:40:44.316560 (+ 2793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:40:44.317231 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:40:44.317239 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:40:44.317242 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:40:44.317382 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:40:44.317715 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:40:44.317718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:40:44.317720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:40:44.318094 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:40:44.318324 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:44.318550 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 23:40:44.318725 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 23:40:44.319883 20195 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:40:44.411233 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 20072
I20260620 23:40:44.428638 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 19711
I20260620 23:40:44.429680 19773 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:40:44.541296 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 19711
2026-06-20T23:40:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4375 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T23:40:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:40:44Z Disabled control of system clock
I20260620 23:40:44.619781 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998525501163-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998525501163-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998525501163-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46001
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:37085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46001
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 23:40:44.961854 20253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:40:44.962186 20253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:40:44.962273 20253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:40:44.969787 20253 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:40:44.969918 20253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:40:44.969957 20253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:40:44.970016 20253 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:40:44.970062 20253 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:40:44.979732 20253 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998525501163-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998525501163-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46001
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998525501163-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:40:44.981093 20253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:40:44.982875 20253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:40:44.995530 20258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:40:44.995615 20259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:40:44.995644 20261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:40:44.996379 20253 server_base.cc:1034] running on GCE node
I20260620 23:40:44.997224 20253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:40:44.998831 20253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:40:45.000075 20253 hybrid_clock.cc:648] HybridClock initialized: now 1781998845000025 us; error 51 us; skew 500 ppm
I20260620 23:40:45.000454 20253 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:40:45.003116 20253 webserver.cc:460] Webserver started at http://127.0.97.62:33981/ using document root <none> and password file <none>
I20260620 23:40:45.003939 20253 fs_manager.cc:359] Metadata directory not provided
I20260620 23:40:45.004083 20253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:40:45.004421 20253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:40:45.007558 20253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998525501163-388-0/minicluster-data/master-0/data/instance:
uuid: "a06d41d6d8f94bb69b2f279c34d98776"
format_stamp: "Formatted at 2026-06-20 23:40:45 on dist-test-slave-tcgx"
server_key: "12e1ba286580fe4141ed5e4408200a9e"
server_key_iv: "4b8a574a42bd1632243cbd46d730fee3"
server_key_version: "encryptionkey@0"
I20260620 23:40:45.008432 20253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998525501163-388-0/minicluster-data/master-0/wal/instance:
uuid: "a06d41d6d8f94bb69b2f279c34d98776"
format_stamp: "Formatted at 2026-06-20 23:40:45 on dist-test-slave-tcgx"
server_key: "12e1ba286580fe4141ed5e4408200a9e"
server_key_iv: "4b8a574a42bd1632243cbd46d730fee3"
server_key_version: "encryptionkey@0"
I20260620 23:40:45.014282 20253 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260620 23:40:45.018170 20267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:45.019747 20253 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:40:45.019971 20253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998525501163-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998525501163-388-0/minicluster-data/master-0/wal
uuid: "a06d41d6d8f94bb69b2f279c34d98776"
format_stamp: "Formatted at 2026-06-20 23:40:45 on dist-test-slave-tcgx"
server_key: "12e1ba286580fe4141ed5e4408200a9e"
server_key_iv: "4b8a574a42bd1632243cbd46d730fee3"
server_key_version: "encryptionkey@0"
I20260620 23:40:45.020226 20253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998525501163-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998525501163-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998525501163-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:40:45.042389 20253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:40:45.043421 20253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:40:45.043743 20253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:40:45.063081 20253 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46001
I20260620 23:40:45.063169 20318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46001 every 8 connection(s)
I20260620 23:40:45.064519 20253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:40:45.065052 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 20253
I20260620 23:40:45.065227 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998525501163-388-0/minicluster-data/master-0/wal/instance
I20260620 23:40:45.065855 388 external_mini_cluster.cc:1428] Setting key 38cb90024faad46b6bc7746e220a20b4
I20260620 23:40:45.069058 20319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:40:45.080355 20322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:45.067880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60460 (local address 127.0.97.62:46001)
0620 23:40:45.068901 (+ 1021us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:45.068914 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:45.068955 (+ 41us) server_negotiation.cc:407] Connection header received
0620 23:40:45.070646 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:45.070670 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:45.071205 (+ 535us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:45.071565 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:40:45.072194 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:45.073571 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:45.075585 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:45.076270 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:45.076416 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:45.077784 (+ 1368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:40:45.077791 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:40:45.077806 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:40:45.078442 (+ 636us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:40:45.078533 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:40:45.078555 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:40:45.078640 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:45.078925 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 23:40:45.079354 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":393,"threads_started":1}
I20260620 23:40:45.081795 20319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a06d41d6d8f94bb69b2f279c34d98776: Bootstrap starting.
I20260620 23:40:45.085635 20319 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a06d41d6d8f94bb69b2f279c34d98776: Neither blocks nor log segments found. Creating new log.
I20260620 23:40:45.086783 20319 log.cc:824] T 00000000000000000000000000000000 P a06d41d6d8f94bb69b2f279c34d98776: Log is configured to *not* fsync() on all Append() calls
I20260620 23:40:45.089591 20319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a06d41d6d8f94bb69b2f279c34d98776: No bootstrap required, opened a new log
I20260620 23:40:45.097110 20319 raft_consensus.cc:348] T 00000000000000000000000000000000 P a06d41d6d8f94bb69b2f279c34d98776 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a06d41d6d8f94bb69b2f279c34d98776" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46001 } }
I20260620 23:40:45.097525 20319 raft_consensus.cc:374] T 00000000000000000000000000000000 P a06d41d6d8f94bb69b2f279c34d98776 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:40:45.097630 20319 raft_consensus.cc:729] T 00000000000000000000000000000000 P a06d41d6d8f94bb69b2f279c34d98776 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a06d41d6d8f94bb69b2f279c34d98776, State: Initialized, Role: FOLLOWER
I20260620 23:40:45.098153 20319 consensus_queue.cc:260] T 00000000000000000000000000000000 P a06d41d6d8f94bb69b2f279c34d98776 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a06d41d6d8f94bb69b2f279c34d98776" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46001 } }
I20260620 23:40:45.098353 20319 raft_consensus.cc:388] T 00000000000000000000000000000000 P a06d41d6d8f94bb69b2f279c34d98776 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:40:45.098452 20319 raft_consensus.cc:482] T 00000000000000000000000000000000 P a06d41d6d8f94bb69b2f279c34d98776 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:40:45.098580 20319 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a06d41d6d8f94bb69b2f279c34d98776 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:40:45.101161 20319 raft_consensus.cc:504] T 00000000000000000000000000000000 P a06d41d6d8f94bb69b2f279c34d98776 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a06d41d6d8f94bb69b2f279c34d98776" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46001 } }
I20260620 23:40:45.101682 20319 leader_election.cc:302] T 00000000000000000000000000000000 P a06d41d6d8f94bb69b2f279c34d98776 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a06d41d6d8f94bb69b2f279c34d98776; no voters:
I20260620 23:40:45.102236 20319 leader_election.cc:288] T 00000000000000000000000000000000 P a06d41d6d8f94bb69b2f279c34d98776 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:40:45.102411 20324 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a06d41d6d8f94bb69b2f279c34d98776 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:40:45.103255 20324 raft_consensus.cc:686] T 00000000000000000000000000000000 P a06d41d6d8f94bb69b2f279c34d98776 [term 1 LEADER]: Becoming Leader. State: Replica: a06d41d6d8f94bb69b2f279c34d98776, State: Running, Role: LEADER
I20260620 23:40:45.103757 20324 consensus_queue.cc:237] T 00000000000000000000000000000000 P a06d41d6d8f94bb69b2f279c34d98776 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a06d41d6d8f94bb69b2f279c34d98776" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46001 } }
I20260620 23:40:45.104470 20319 sys_catalog.cc:564] T 00000000000000000000000000000000 P a06d41d6d8f94bb69b2f279c34d98776 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:40:45.106840 20325 sys_catalog.cc:455] T 00000000000000000000000000000000 P a06d41d6d8f94bb69b2f279c34d98776 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a06d41d6d8f94bb69b2f279c34d98776" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a06d41d6d8f94bb69b2f279c34d98776" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46001 } } }
I20260620 23:40:45.106917 20326 sys_catalog.cc:455] T 00000000000000000000000000000000 P a06d41d6d8f94bb69b2f279c34d98776 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a06d41d6d8f94bb69b2f279c34d98776. Latest consensus state: current_term: 1 leader_uuid: "a06d41d6d8f94bb69b2f279c34d98776" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a06d41d6d8f94bb69b2f279c34d98776" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46001 } } }
I20260620 23:40:45.107312 20325 sys_catalog.cc:458] T 00000000000000000000000000000000 P a06d41d6d8f94bb69b2f279c34d98776 [sys.catalog]: This master's current role is: LEADER
I20260620 23:40:45.107378 20326 sys_catalog.cc:458] T 00000000000000000000000000000000 P a06d41d6d8f94bb69b2f279c34d98776 [sys.catalog]: This master's current role is: LEADER
I20260620 23:40:45.109843 20331 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:40:45.115310 20331 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:40:45.124970 20331 catalog_manager.cc:1347] Generated new cluster ID: ab85ae89867140a29a2c7fd65444e674
I20260620 23:40:45.125155 20331 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:40:45.151062 20331 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:40:45.152415 20331 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:40:45.158198 20331 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a06d41d6d8f94bb69b2f279c34d98776: Generated new TSK 0
I20260620 23:40:45.159394 20331 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:40:45.164537 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998525501163-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998525501163-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998525501163-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46001
--builtin_ntp_servers=127.0.97.20:37085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 23:40:45.531093 20343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:40:45.531442 20343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:40:45.531621 20343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:40:45.539243 20343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:40:45.539405 20343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:40:45.539585 20343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 23:40:45.549206 20343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998525501163-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998525501163-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46001
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998525501163-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:40:45.550571 20343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:40:45.552501 20343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:40:45.566159 20349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:40:45.566226 20351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:40:45.566192 20348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:40:45.567072 20343 server_base.cc:1034] running on GCE node
I20260620 23:40:45.567709 20343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:40:45.568763 20343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:40:45.570005 20343 hybrid_clock.cc:648] HybridClock initialized: now 1781998845569950 us; error 52 us; skew 500 ppm
I20260620 23:40:45.570343 20343 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:40:45.576823 20343 webserver.cc:460] Webserver started at http://127.0.97.1:39077/ using document root <none> and password file <none>
I20260620 23:40:45.577639 20343 fs_manager.cc:359] Metadata directory not provided
I20260620 23:40:45.577783 20343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:40:45.578131 20343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:40:45.581264 20343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998525501163-388-0/minicluster-data/ts-0/data/instance:
uuid: "361c8462026b48e38e690a913d4d8808"
format_stamp: "Formatted at 2026-06-20 23:40:45 on dist-test-slave-tcgx"
server_key: "152adfc2fb6271d7edc943aaa18f6e06"
server_key_iv: "54c1c6329fe1a384ee5d74f1aaaf3b7d"
server_key_version: "encryptionkey@0"
I20260620 23:40:45.582275 20343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998525501163-388-0/minicluster-data/ts-0/wal/instance:
uuid: "361c8462026b48e38e690a913d4d8808"
format_stamp: "Formatted at 2026-06-20 23:40:45 on dist-test-slave-tcgx"
server_key: "152adfc2fb6271d7edc943aaa18f6e06"
server_key_iv: "54c1c6329fe1a384ee5d74f1aaaf3b7d"
server_key_version: "encryptionkey@0"
I20260620 23:40:45.588147 20343 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 23:40:45.591943 20357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:45.593468 20343 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 23:40:45.593618 20343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998525501163-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998525501163-388-0/minicluster-data/ts-0/wal
uuid: "361c8462026b48e38e690a913d4d8808"
format_stamp: "Formatted at 2026-06-20 23:40:45 on dist-test-slave-tcgx"
server_key: "152adfc2fb6271d7edc943aaa18f6e06"
server_key_iv: "54c1c6329fe1a384ee5d74f1aaaf3b7d"
server_key_version: "encryptionkey@0"
I20260620 23:40:45.593827 20343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998525501163-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998525501163-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998525501163-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:40:45.629223 20343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:40:45.630066 20343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:40:45.630306 20343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:40:45.631309 20343 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:40:45.632896 20343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:40:45.632982 20343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:45.633071 20343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:40:45.633126 20343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:45.667084 20343 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45065
I20260620 23:40:45.667186 20469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45065 every 8 connection(s)
I20260620 23:40:45.668448 20343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998525501163-388-0/minicluster-data/ts-0/data/info.pb
I20260620 23:40:45.673883 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 20343
I20260620 23:40:45.674058 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998525501163-388-0/minicluster-data/ts-0/wal/instance
I20260620 23:40:45.674597 388 external_mini_cluster.cc:1428] Setting key 3f00f5e8d1485bfdc7e369808ba5442c
I20260620 23:40:45.677667 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998525501163-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998525501163-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998525501163-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46001
--builtin_ntp_servers=127.0.97.20:37085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 23:40:45.682003 20472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:45.671112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60605 (local address 127.0.97.62:46001)
0620 23:40:45.671582 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:45.671587 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:45.672667 (+ 1080us) server_negotiation.cc:407] Connection header received
0620 23:40:45.675098 (+ 2431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:45.675103 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:45.675174 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:45.675267 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:40:45.676600 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:45.677348 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:45.679093 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:45.679494 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:45.679636 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:45.681155 (+ 1519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:40:45.681159 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:40:45.681162 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:40:45.681181 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:40:45.681219 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:40:45.681231 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:40:45.681321 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:45.681554 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 23:40:45.681657 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":254,"threads_started":1}
I20260620 23:40:45.683140 20473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:45.671600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46001 (local address 127.0.97.1:60605)
0620 23:40:45.672504 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:45.672539 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 23:40:45.674735 (+ 2196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:40:45.675523 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:40:45.675531 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:40:45.675627 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:40:45.676415 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:45.676441 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:45.677505 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:45.677510 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:45.678867 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:45.678879 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:45.679742 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:45.679750 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:45.680036 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:45.680804 (+ 768us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:40:45.680807 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:40:45.680846 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:40:45.680930 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:40:45.681337 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:40:45.681351 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:40:45.681363 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 23:40:45.681546 (+ 183us) client_negotiation.cc:240] Negotiation successful
0620 23:40:45.681966 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":275,"threads_started":1}
I20260620 23:40:45.685783 20470 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46001
I20260620 23:40:45.686188 20470 heartbeater.cc:464] Registering TS with master...
I20260620 23:40:45.687055 20470 heartbeater.cc:511] Master 127.0.97.62:46001 requested a full tablet report, sending...
I20260620 23:40:45.689258 20284 ts_manager.cc:194] Registered new tserver with Master: 361c8462026b48e38e690a913d4d8808 (127.0.97.1:45065)
I20260620 23:40:45.690757 20284 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:60605
W20260620 23:40:46.017303 20474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:40:46.017613 20474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:40:46.017765 20474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:40:46.025439 20474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:40:46.025604 20474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:40:46.025799 20474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 23:40:46.035480 20474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998525501163-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998525501163-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46001
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998525501163-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:40:46.036860 20474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:40:46.038662 20474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:40:46.051956 20479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:40:46.052099 20480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:40:46.052078 20482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:40:46.052848 20474 server_base.cc:1034] running on GCE node
I20260620 23:40:46.053493 20474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:40:46.054494 20474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:40:46.055739 20474 hybrid_clock.cc:648] HybridClock initialized: now 1781998846055684 us; error 47 us; skew 500 ppm
I20260620 23:40:46.056133 20474 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:40:46.058523 20474 webserver.cc:460] Webserver started at http://127.0.97.2:33887/ using document root <none> and password file <none>
I20260620 23:40:46.059334 20474 fs_manager.cc:359] Metadata directory not provided
I20260620 23:40:46.059474 20474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:40:46.059824 20474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:40:46.063068 20474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998525501163-388-0/minicluster-data/ts-1/data/instance:
uuid: "6c4cc9e5cbde428e924eb7d8013ecc99"
format_stamp: "Formatted at 2026-06-20 23:40:46 on dist-test-slave-tcgx"
server_key: "e19b9a0cd981470a48884f1b35544fae"
server_key_iv: "5ee6588e97be58014128950560ab810e"
server_key_version: "encryptionkey@0"
I20260620 23:40:46.064105 20474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998525501163-388-0/minicluster-data/ts-1/wal/instance:
uuid: "6c4cc9e5cbde428e924eb7d8013ecc99"
format_stamp: "Formatted at 2026-06-20 23:40:46 on dist-test-slave-tcgx"
server_key: "e19b9a0cd981470a48884f1b35544fae"
server_key_iv: "5ee6588e97be58014128950560ab810e"
server_key_version: "encryptionkey@0"
I20260620 23:40:46.071034 20474 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260620 23:40:46.075482 20488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:46.077101 20474 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260620 23:40:46.077276 20474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998525501163-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998525501163-388-0/minicluster-data/ts-1/wal
uuid: "6c4cc9e5cbde428e924eb7d8013ecc99"
format_stamp: "Formatted at 2026-06-20 23:40:46 on dist-test-slave-tcgx"
server_key: "e19b9a0cd981470a48884f1b35544fae"
server_key_iv: "5ee6588e97be58014128950560ab810e"
server_key_version: "encryptionkey@0"
I20260620 23:40:46.077523 20474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998525501163-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998525501163-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998525501163-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:40:46.094278 20474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:40:46.095211 20474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:40:46.095517 20474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:40:46.096427 20474 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:40:46.098078 20474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:40:46.098165 20474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:46.098289 20474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:40:46.098348 20474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:46.133279 20474 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43295
I20260620 23:40:46.133322 20600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43295 every 8 connection(s)
I20260620 23:40:46.134743 20474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998525501163-388-0/minicluster-data/ts-1/data/info.pb
I20260620 23:40:46.144796 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 20474
I20260620 23:40:46.145007 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998525501163-388-0/minicluster-data/ts-1/wal/instance
I20260620 23:40:46.145426 388 external_mini_cluster.cc:1428] Setting key cbb1b026f3ab6d2062a265311f7e6584
I20260620 23:40:46.148125 20472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:46.137504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34965 (local address 127.0.97.62:46001)
0620 23:40:46.137696 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:46.137700 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:46.139184 (+ 1484us) server_negotiation.cc:407] Connection header received
0620 23:40:46.141003 (+ 1819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:46.141008 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:46.141101 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:46.141213 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:40:46.142311 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:46.143124 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:46.144949 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:46.145451 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:46.145608 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:46.147188 (+ 1580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:40:46.147193 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:40:46.147197 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:40:46.147216 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:40:46.147246 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:40:46.147257 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:40:46.147380 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:46.147697 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 23:40:46.147812 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 23:40:46.148522 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998525501163-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998525501163-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998525501163-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46001
--builtin_ntp_servers=127.0.97.20:37085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 23:40:46.149176 20603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:46.138017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46001 (local address 127.0.97.2:34965)
0620 23:40:46.138998 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:46.139063 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 23:40:46.140713 (+ 1650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:40:46.141457 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:40:46.141466 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:40:46.141558 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:40:46.142143 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:46.142160 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:46.143270 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:46.143276 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:46.144722 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:46.144737 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:46.145712 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:46.145719 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:46.146020 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:46.146804 (+ 784us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:40:46.146807 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:40:46.146848 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:40:46.146928 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:40:46.147455 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:40:46.147467 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:40:46.147476 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 23:40:46.147652 (+ 176us) client_negotiation.cc:240] Negotiation successful
0620 23:40:46.147972 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":290,"threads_started":1}
I20260620 23:40:46.150661 20601 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46001
I20260620 23:40:46.151011 20601 heartbeater.cc:464] Registering TS with master...
I20260620 23:40:46.151945 20601 heartbeater.cc:511] Master 127.0.97.62:46001 requested a full tablet report, sending...
I20260620 23:40:46.153549 20284 ts_manager.cc:194] Registered new tserver with Master: 6c4cc9e5cbde428e924eb7d8013ecc99 (127.0.97.2:43295)
I20260620 23:40:46.154382 20284 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:34965
W20260620 23:40:46.484607 20604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:40:46.484905 20604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:40:46.485056 20604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:40:46.492298 20604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:40:46.492406 20604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:40:46.492604 20604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 23:40:46.501956 20604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998525501163-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998525501163-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46001
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998525501163-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:40:46.503341 20604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:40:46.505228 20604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:40:46.519467 20609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:40:46.519852 20610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:40:46.519497 20612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:40:46.520728 20604 server_base.cc:1034] running on GCE node
I20260620 23:40:46.521395 20604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:40:46.522410 20604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:40:46.523649 20604 hybrid_clock.cc:648] HybridClock initialized: now 1781998846523602 us; error 49 us; skew 500 ppm
I20260620 23:40:46.524106 20604 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:40:46.526468 20604 webserver.cc:460] Webserver started at http://127.0.97.3:42337/ using document root <none> and password file <none>
I20260620 23:40:46.527263 20604 fs_manager.cc:359] Metadata directory not provided
I20260620 23:40:46.527393 20604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:40:46.527717 20604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:40:46.530645 20604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998525501163-388-0/minicluster-data/ts-2/data/instance:
uuid: "1854b4080cd44c4aaa0b98fd72c2da2b"
format_stamp: "Formatted at 2026-06-20 23:40:46 on dist-test-slave-tcgx"
server_key: "ae711d352d5008723711c80387a3ef70"
server_key_iv: "90fb5f1912dd1a5e3f258b32a76ad65b"
server_key_version: "encryptionkey@0"
I20260620 23:40:46.531571 20604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998525501163-388-0/minicluster-data/ts-2/wal/instance:
uuid: "1854b4080cd44c4aaa0b98fd72c2da2b"
format_stamp: "Formatted at 2026-06-20 23:40:46 on dist-test-slave-tcgx"
server_key: "ae711d352d5008723711c80387a3ef70"
server_key_iv: "90fb5f1912dd1a5e3f258b32a76ad65b"
server_key_version: "encryptionkey@0"
I20260620 23:40:46.536919 20604 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:40:46.540581 20618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:46.541993 20604 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 23:40:46.542285 20604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998525501163-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998525501163-388-0/minicluster-data/ts-2/wal
uuid: "1854b4080cd44c4aaa0b98fd72c2da2b"
format_stamp: "Formatted at 2026-06-20 23:40:46 on dist-test-slave-tcgx"
server_key: "ae711d352d5008723711c80387a3ef70"
server_key_iv: "90fb5f1912dd1a5e3f258b32a76ad65b"
server_key_version: "encryptionkey@0"
I20260620 23:40:46.542537 20604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998525501163-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998525501163-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998525501163-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:40:46.587656 20604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:40:46.588589 20604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:40:46.588876 20604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:40:46.589916 20604 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:40:46.591563 20604 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:40:46.591684 20604 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:46.591822 20604 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:40:46.591909 20604 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:46.626533 20604 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43129
I20260620 23:40:46.626583 20730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43129 every 8 connection(s)
I20260620 23:40:46.627933 20604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998525501163-388-0/minicluster-data/ts-2/data/info.pb
I20260620 23:40:46.636381 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 20604
I20260620 23:40:46.636654 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998525501163-388-0/minicluster-data/ts-2/wal/instance
I20260620 23:40:46.637161 388 external_mini_cluster.cc:1428] Setting key 845b371f077a22581d3be229ad89c55a
I20260620 23:40:46.641327 20472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:46.630665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52351 (local address 127.0.97.62:46001)
0620 23:40:46.630887 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:46.630894 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:46.632422 (+ 1528us) server_negotiation.cc:407] Connection header received
0620 23:40:46.634220 (+ 1798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:46.634224 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:46.634307 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:46.634402 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:40:46.635597 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:46.636393 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:46.638247 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:46.638934 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:46.639087 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:46.640638 (+ 1551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:40:46.640644 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:40:46.640647 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:40:46.640665 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:40:46.640695 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:40:46.640706 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:40:46.640776 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:46.641008 (+ 232us) server_negotiation.cc:299] Negotiation successful
0620 23:40:46.641084 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 23:40:46.642360 20733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:46.631253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46001 (local address 127.0.97.3:52351)
0620 23:40:46.632228 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:46.632271 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 23:40:46.633925 (+ 1654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:40:46.634619 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:40:46.634627 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:40:46.634720 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:40:46.635428 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:46.635447 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:46.636561 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:46.636566 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:46.638107 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:46.638125 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:46.639182 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:46.639189 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:46.639501 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:46.640314 (+ 813us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:40:46.640317 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:40:46.640357 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:40:46.640446 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:40:46.640793 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:40:46.640801 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:40:46.640807 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 23:40:46.640981 (+ 174us) client_negotiation.cc:240] Negotiation successful
0620 23:40:46.641278 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":311,"threads_started":1}
I20260620 23:40:46.643764 20731 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46001
I20260620 23:40:46.644039 20731 heartbeater.cc:464] Registering TS with master...
I20260620 23:40:46.644778 20731 heartbeater.cc:511] Master 127.0.97.62:46001 requested a full tablet report, sending...
I20260620 23:40:46.646108 20284 ts_manager.cc:194] Registered new tserver with Master: 1854b4080cd44c4aaa0b98fd72c2da2b (127.0.97.3:43129)
I20260620 23:40:46.646917 20284 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:52351
I20260620 23:40:46.652422 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:40:46.661837 20472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:46.656422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51090 (local address 127.0.97.62:46001)
0620 23:40:46.656643 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:46.656653 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:46.657013 (+ 360us) server_negotiation.cc:407] Connection header received
0620 23:40:46.657134 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:46.657137 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:46.657206 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:46.657288 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:40:46.657712 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:46.658375 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:46.659771 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:46.660168 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:46.660262 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:46.661203 (+ 941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:40:46.661207 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:40:46.661211 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:40:46.661227 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:40:46.661258 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:40:46.661269 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:40:46.661336 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:46.661452 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 23:40:46.661521 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 23:40:46.665454 20284 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51090:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:40:46.672199 20284 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:40:46.693691 20470 heartbeater.cc:503] Master 127.0.97.62:46001 was elected leader, sending a full tablet report...
I20260620 23:40:46.701435 20744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:46.684413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43129 (local address 127.0.0.1:53454)
0620 23:40:46.686629 (+ 2216us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:46.686666 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 23:40:46.686884 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:40:46.688133 (+ 1249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:40:46.688144 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:40:46.688186 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:40:46.688693 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:46.688709 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:46.691352 (+ 2643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:46.691361 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:46.697806 (+ 6445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:46.697833 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:46.700454 (+ 2621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:46.700460 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:46.700680 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:46.700733 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 23:40:46.700858 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 23:40:46.701092 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1925,"thread_start_us":1989,"threads_started":1}
I20260620 23:40:46.702023 20743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:46.684844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45065 (local address 127.0.0.1:53104)
0620 23:40:46.686686 (+ 1842us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:46.686705 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:40:46.686884 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:40:46.689469 (+ 2585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:40:46.689477 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:40:46.689513 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:40:46.690029 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:46.690043 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:46.693928 (+ 3885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:46.693933 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:46.698748 (+ 4815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:46.698769 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:46.701112 (+ 2343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:46.701121 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:46.701326 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:46.701355 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 23:40:46.701455 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 23:40:46.701609 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1713,"mutex_wait_us":1,"thread_start_us":1354,"threads_started":1}
I20260620 23:40:46.702423 20745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:46.684842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53454 (local address 127.0.97.3:43129)
0620 23:40:46.685646 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:46.685660 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:46.686778 (+ 1118us) server_negotiation.cc:407] Connection header received
0620 23:40:46.687171 (+ 393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:46.687181 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:46.687472 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:46.687670 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:40:46.688964 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:46.691119 (+ 2155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:46.698117 (+ 6998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:46.700229 (+ 2112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:46.700419 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:46.701841 (+ 1422us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:46.701930 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 23:40:46.702129 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":436,"threads_started":1}
I20260620 23:40:46.702896 20746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:46.685384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53104 (local address 127.0.97.1:45065)
0620 23:40:46.686212 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:46.686223 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:46.686974 (+ 751us) server_negotiation.cc:407] Connection header received
0620 23:40:46.688578 (+ 1604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:46.688592 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:46.688933 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:46.689157 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:40:46.690258 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:46.693747 (+ 3489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:46.699067 (+ 5320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:46.700601 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:46.700719 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:46.700973 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:46.702357 (+ 1384us) server_negotiation.cc:299] Negotiation successful
0620 23:40:46.702546 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":421,"threads_started":1}
I20260620 23:40:46.706703 20405 tablet_service.cc:1467] Processing CreateTablet for tablet fbfb2c018e1e41df85c0c9e0d454f18f (DEFAULT_TABLE table=test-table [id=cce57cc1123a4468afced481f51b1602]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:40:46.706703 20666 tablet_service.cc:1467] Processing CreateTablet for tablet fbfb2c018e1e41df85c0c9e0d454f18f (DEFAULT_TABLE table=test-table [id=cce57cc1123a4468afced481f51b1602]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:40:46.708006 20747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:46.687126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43295 (local address 127.0.0.1:33510)
0620 23:40:46.700195 (+ 13069us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:46.700215 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:40:46.700442 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:40:46.701468 (+ 1026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:40:46.701472 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:40:46.701497 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:40:46.701855 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:46.701866 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:46.703816 (+ 1950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:46.703822 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:46.705381 (+ 1559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:46.705401 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:46.707211 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:46.707218 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:46.707399 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:46.707432 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 23:40:46.707518 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 23:40:46.707650 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12946,"thread_start_us":12815,"threads_started":1}
I20260620 23:40:46.708397 20405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbfb2c018e1e41df85c0c9e0d454f18f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:40:46.708395 20666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbfb2c018e1e41df85c0c9e0d454f18f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:40:46.708815 20748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:46.687359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33510 (local address 127.0.97.2:43295)
0620 23:40:46.699602 (+ 12243us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:46.699612 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:46.700319 (+ 707us) server_negotiation.cc:407] Connection header received
0620 23:40:46.700681 (+ 362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:46.700690 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:46.700991 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:46.701194 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:40:46.702092 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:46.703665 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:46.705569 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:46.706664 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:46.706797 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:46.707071 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:46.708376 (+ 1305us) server_negotiation.cc:299] Negotiation successful
0620 23:40:46.708556 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12061,"thread_start_us":12253,"threads_started":1}
I20260620 23:40:46.714184 20536 tablet_service.cc:1467] Processing CreateTablet for tablet fbfb2c018e1e41df85c0c9e0d454f18f (DEFAULT_TABLE table=test-table [id=cce57cc1123a4468afced481f51b1602]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:40:46.715627 20536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbfb2c018e1e41df85c0c9e0d454f18f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:40:46.725709 20749 tablet_bootstrap.cc:492] T fbfb2c018e1e41df85c0c9e0d454f18f P 361c8462026b48e38e690a913d4d8808: Bootstrap starting.
I20260620 23:40:46.726137 20750 tablet_bootstrap.cc:492] T fbfb2c018e1e41df85c0c9e0d454f18f P 1854b4080cd44c4aaa0b98fd72c2da2b: Bootstrap starting.
I20260620 23:40:46.729566 20751 tablet_bootstrap.cc:492] T fbfb2c018e1e41df85c0c9e0d454f18f P 6c4cc9e5cbde428e924eb7d8013ecc99: Bootstrap starting.
I20260620 23:40:46.729635 20749 tablet_bootstrap.cc:654] T fbfb2c018e1e41df85c0c9e0d454f18f P 361c8462026b48e38e690a913d4d8808: Neither blocks nor log segments found. Creating new log.
I20260620 23:40:46.730019 20750 tablet_bootstrap.cc:654] T fbfb2c018e1e41df85c0c9e0d454f18f P 1854b4080cd44c4aaa0b98fd72c2da2b: Neither blocks nor log segments found. Creating new log.
I20260620 23:40:46.730966 20749 log.cc:824] T fbfb2c018e1e41df85c0c9e0d454f18f P 361c8462026b48e38e690a913d4d8808: Log is configured to *not* fsync() on all Append() calls
I20260620 23:40:46.731196 20750 log.cc:824] T fbfb2c018e1e41df85c0c9e0d454f18f P 1854b4080cd44c4aaa0b98fd72c2da2b: Log is configured to *not* fsync() on all Append() calls
I20260620 23:40:46.733156 20751 tablet_bootstrap.cc:654] T fbfb2c018e1e41df85c0c9e0d454f18f P 6c4cc9e5cbde428e924eb7d8013ecc99: Neither blocks nor log segments found. Creating new log.
I20260620 23:40:46.734216 20749 tablet_bootstrap.cc:492] T fbfb2c018e1e41df85c0c9e0d454f18f P 361c8462026b48e38e690a913d4d8808: No bootstrap required, opened a new log
I20260620 23:40:46.734216 20750 tablet_bootstrap.cc:492] T fbfb2c018e1e41df85c0c9e0d454f18f P 1854b4080cd44c4aaa0b98fd72c2da2b: No bootstrap required, opened a new log
I20260620 23:40:46.734223 20751 log.cc:824] T fbfb2c018e1e41df85c0c9e0d454f18f P 6c4cc9e5cbde428e924eb7d8013ecc99: Log is configured to *not* fsync() on all Append() calls
I20260620 23:40:46.734494 20749 ts_tablet_manager.cc:1397] T fbfb2c018e1e41df85c0c9e0d454f18f P 361c8462026b48e38e690a913d4d8808: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260620 23:40:46.734494 20750 ts_tablet_manager.cc:1397] T fbfb2c018e1e41df85c0c9e0d454f18f P 1854b4080cd44c4aaa0b98fd72c2da2b: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260620 23:40:46.737798 20751 tablet_bootstrap.cc:492] T fbfb2c018e1e41df85c0c9e0d454f18f P 6c4cc9e5cbde428e924eb7d8013ecc99: No bootstrap required, opened a new log
I20260620 23:40:46.738090 20751 ts_tablet_manager.cc:1397] T fbfb2c018e1e41df85c0c9e0d454f18f P 6c4cc9e5cbde428e924eb7d8013ecc99: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260620 23:40:46.745683 20749 raft_consensus.cc:348] T fbfb2c018e1e41df85c0c9e0d454f18f P 361c8462026b48e38e690a913d4d8808 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1854b4080cd44c4aaa0b98fd72c2da2b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43129 } } peers { permanent_uuid: "361c8462026b48e38e690a913d4d8808" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45065 } } peers { permanent_uuid: "6c4cc9e5cbde428e924eb7d8013ecc99" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43295 } }
I20260620 23:40:46.745685 20750 raft_consensus.cc:348] T fbfb2c018e1e41df85c0c9e0d454f18f P 1854b4080cd44c4aaa0b98fd72c2da2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1854b4080cd44c4aaa0b98fd72c2da2b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43129 } } peers { permanent_uuid: "361c8462026b48e38e690a913d4d8808" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45065 } } peers { permanent_uuid: "6c4cc9e5cbde428e924eb7d8013ecc99" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43295 } }
I20260620 23:40:46.745846 20751 raft_consensus.cc:348] T fbfb2c018e1e41df85c0c9e0d454f18f P 6c4cc9e5cbde428e924eb7d8013ecc99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1854b4080cd44c4aaa0b98fd72c2da2b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43129 } } peers { permanent_uuid: "361c8462026b48e38e690a913d4d8808" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45065 } } peers { permanent_uuid: "6c4cc9e5cbde428e924eb7d8013ecc99" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43295 } }
I20260620 23:40:46.746217 20749 raft_consensus.cc:374] T fbfb2c018e1e41df85c0c9e0d454f18f P 361c8462026b48e38e690a913d4d8808 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:40:46.746217 20750 raft_consensus.cc:374] T fbfb2c018e1e41df85c0c9e0d454f18f P 1854b4080cd44c4aaa0b98fd72c2da2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:40:46.746313 20749 raft_consensus.cc:729] T fbfb2c018e1e41df85c0c9e0d454f18f P 361c8462026b48e38e690a913d4d8808 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 361c8462026b48e38e690a913d4d8808, State: Initialized, Role: FOLLOWER
I20260620 23:40:46.746354 20751 raft_consensus.cc:374] T fbfb2c018e1e41df85c0c9e0d454f18f P 6c4cc9e5cbde428e924eb7d8013ecc99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:40:46.746377 20750 raft_consensus.cc:729] T fbfb2c018e1e41df85c0c9e0d454f18f P 1854b4080cd44c4aaa0b98fd72c2da2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1854b4080cd44c4aaa0b98fd72c2da2b, State: Initialized, Role: FOLLOWER
I20260620 23:40:46.746474 20751 raft_consensus.cc:729] T fbfb2c018e1e41df85c0c9e0d454f18f P 6c4cc9e5cbde428e924eb7d8013ecc99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c4cc9e5cbde428e924eb7d8013ecc99, State: Initialized, Role: FOLLOWER
I20260620 23:40:46.747068 20749 consensus_queue.cc:260] T fbfb2c018e1e41df85c0c9e0d454f18f P 361c8462026b48e38e690a913d4d8808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1854b4080cd44c4aaa0b98fd72c2da2b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43129 } } peers { permanent_uuid: "361c8462026b48e38e690a913d4d8808" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45065 } } peers { permanent_uuid: "6c4cc9e5cbde428e924eb7d8013ecc99" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43295 } }
I20260620 23:40:46.747068 20750 consensus_queue.cc:260] T fbfb2c018e1e41df85c0c9e0d454f18f P 1854b4080cd44c4aaa0b98fd72c2da2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1854b4080cd44c4aaa0b98fd72c2da2b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43129 } } peers { permanent_uuid: "361c8462026b48e38e690a913d4d8808" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45065 } } peers { permanent_uuid: "6c4cc9e5cbde428e924eb7d8013ecc99" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43295 } }
I20260620 23:40:46.747289 20751 consensus_queue.cc:260] T fbfb2c018e1e41df85c0c9e0d454f18f P 6c4cc9e5cbde428e924eb7d8013ecc99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1854b4080cd44c4aaa0b98fd72c2da2b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43129 } } peers { permanent_uuid: "361c8462026b48e38e690a913d4d8808" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45065 } } peers { permanent_uuid: "6c4cc9e5cbde428e924eb7d8013ecc99" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43295 } }
I20260620 23:40:46.750216 20601 heartbeater.cc:503] Master 127.0.97.62:46001 was elected leader, sending a full tablet report...
I20260620 23:40:46.751012 20751 ts_tablet_manager.cc:1428] T fbfb2c018e1e41df85c0c9e0d454f18f P 6c4cc9e5cbde428e924eb7d8013ecc99: Time spent starting tablet: real 0.013s user 0.009s sys 0.006s
I20260620 23:40:46.752882 20731 heartbeater.cc:503] Master 127.0.97.62:46001 was elected leader, sending a full tablet report...
I20260620 23:40:46.753562 20750 ts_tablet_manager.cc:1428] T fbfb2c018e1e41df85c0c9e0d454f18f P 1854b4080cd44c4aaa0b98fd72c2da2b: Time spent starting tablet: real 0.019s user 0.012s sys 0.008s
I20260620 23:40:46.755457 20749 ts_tablet_manager.cc:1428] T fbfb2c018e1e41df85c0c9e0d454f18f P 361c8462026b48e38e690a913d4d8808: Time spent starting tablet: real 0.021s user 0.011s sys 0.007s
I20260620 23:40:46.846607 20755 raft_consensus.cc:482] T fbfb2c018e1e41df85c0c9e0d454f18f P 361c8462026b48e38e690a913d4d8808 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:40:46.847031 20755 raft_consensus.cc:504] T fbfb2c018e1e41df85c0c9e0d454f18f P 361c8462026b48e38e690a913d4d8808 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1854b4080cd44c4aaa0b98fd72c2da2b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43129 } } peers { permanent_uuid: "361c8462026b48e38e690a913d4d8808" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45065 } } peers { permanent_uuid: "6c4cc9e5cbde428e924eb7d8013ecc99" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43295 } }
I20260620 23:40:46.848763 20755 leader_election.cc:288] T fbfb2c018e1e41df85c0c9e0d454f18f P 361c8462026b48e38e690a913d4d8808 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1854b4080cd44c4aaa0b98fd72c2da2b (127.0.97.3:43129), 6c4cc9e5cbde428e924eb7d8013ecc99 (127.0.97.2:43295)
I20260620 23:40:46.852979 20757 raft_consensus.cc:482] T fbfb2c018e1e41df85c0c9e0d454f18f P 6c4cc9e5cbde428e924eb7d8013ecc99 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:40:46.853293 20757 raft_consensus.cc:504] T fbfb2c018e1e41df85c0c9e0d454f18f P 6c4cc9e5cbde428e924eb7d8013ecc99 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1854b4080cd44c4aaa0b98fd72c2da2b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43129 } } peers { permanent_uuid: "361c8462026b48e38e690a913d4d8808" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45065 } } peers { permanent_uuid: "6c4cc9e5cbde428e924eb7d8013ecc99" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43295 } }
I20260620 23:40:46.855571 20757 leader_election.cc:288] T fbfb2c018e1e41df85c0c9e0d454f18f P 6c4cc9e5cbde428e924eb7d8013ecc99 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1854b4080cd44c4aaa0b98fd72c2da2b (127.0.97.3:43129), 361c8462026b48e38e690a913d4d8808 (127.0.97.1:45065)
I20260620 23:40:46.858137 20745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:46.849314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48069 (local address 127.0.97.3:43129)
0620 23:40:46.849542 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:46.849552 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:46.850095 (+ 543us) server_negotiation.cc:407] Connection header received
0620 23:40:46.850325 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:46.850334 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:46.850447 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:46.850546 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:40:46.851210 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:46.851930 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:46.854399 (+ 2469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:46.855625 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:46.855763 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:46.855877 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:46.857572 (+ 1695us) server_negotiation.cc:299] Negotiation successful
0620 23:40:46.857779 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 23:40:46.858457 20758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:46.849262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43129 (local address 127.0.97.1:48069)
0620 23:40:46.849933 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:46.849949 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:40:46.850149 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:40:46.850615 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:40:46.850620 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:40:46.850658 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:40:46.851053 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:46.851063 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:46.852064 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:46.852072 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:46.854092 (+ 2020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:46.854112 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:46.857030 (+ 2918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:46.857040 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:46.857297 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:46.857378 (+ 81us) client_negotiation.cc:769] Sending connection context
0620 23:40:46.857497 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 23:40:46.857702 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"mutex_wait_us":41,"thread_start_us":376,"threads_started":1}
I20260620 23:40:46.860009 20686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbfb2c018e1e41df85c0c9e0d454f18f" candidate_uuid: "361c8462026b48e38e690a913d4d8808" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1854b4080cd44c4aaa0b98fd72c2da2b" is_pre_election: true
I20260620 23:40:46.860858 20686 raft_consensus.cc:2454] T fbfb2c018e1e41df85c0c9e0d454f18f P 1854b4080cd44c4aaa0b98fd72c2da2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 361c8462026b48e38e690a913d4d8808 in term 0.
I20260620 23:40:46.861994 20360 leader_election.cc:302] T fbfb2c018e1e41df85c0c9e0d454f18f P 361c8462026b48e38e690a913d4d8808 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1854b4080cd44c4aaa0b98fd72c2da2b, 361c8462026b48e38e690a913d4d8808; no voters:
I20260620 23:40:46.862507 20755 raft_consensus.cc:2781] T fbfb2c018e1e41df85c0c9e0d454f18f P 361c8462026b48e38e690a913d4d8808 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:40:46.862784 20755 raft_consensus.cc:482] T fbfb2c018e1e41df85c0c9e0d454f18f P 361c8462026b48e38e690a913d4d8808 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:40:46.862895 20748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:46.850252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43927 (local address 127.0.97.2:43295)
0620 23:40:46.850472 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:46.850479 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:46.850790 (+ 311us) server_negotiation.cc:407] Connection header received
0620 23:40:46.851021 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:46.851033 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:46.851103 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:46.851207 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:40:46.851773 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:46.852439 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:46.854649 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:46.855897 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:46.856052 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:46.856140 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:46.862484 (+ 6344us) server_negotiation.cc:299] Negotiation successful
0620 23:40:46.862625 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 23:40:46.863013 20759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:46.850113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43295 (local address 127.0.97.1:43927)
0620 23:40:46.850685 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:46.850698 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 23:40:46.850858 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:40:46.851224 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:40:46.851229 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:40:46.851252 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:40:46.851616 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:46.851626 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:46.852591 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:46.852598 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:46.854453 (+ 1855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:46.854467 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:46.856082 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:46.856088 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:46.862300 (+ 6212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:46.862341 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 23:40:46.862447 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 23:40:46.862599 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"mutex_wait_us":27,"thread_start_us":267,"threads_started":1}
I20260620 23:40:46.863291 20755 raft_consensus.cc:3037] T fbfb2c018e1e41df85c0c9e0d454f18f P 361c8462026b48e38e690a913d4d8808 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:40:46.864135 20556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbfb2c018e1e41df85c0c9e0d454f18f" candidate_uuid: "361c8462026b48e38e690a913d4d8808" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c4cc9e5cbde428e924eb7d8013ecc99" is_pre_election: true
I20260620 23:40:46.864588 20556 raft_consensus.cc:2454] T fbfb2c018e1e41df85c0c9e0d454f18f P 6c4cc9e5cbde428e924eb7d8013ecc99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 361c8462026b48e38e690a913d4d8808 in term 0.
I20260620 23:40:46.867297 20755 raft_consensus.cc:504] T fbfb2c018e1e41df85c0c9e0d454f18f P 361c8462026b48e38e690a913d4d8808 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1854b4080cd44c4aaa0b98fd72c2da2b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43129 } } peers { permanent_uuid: "361c8462026b48e38e690a913d4d8808" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45065 } } peers { permanent_uuid: "6c4cc9e5cbde428e924eb7d8013ecc99" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43295 } }
I20260620 23:40:46.868752 20686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbfb2c018e1e41df85c0c9e0d454f18f" candidate_uuid: "361c8462026b48e38e690a913d4d8808" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1854b4080cd44c4aaa0b98fd72c2da2b"
I20260620 23:40:46.869094 20686 raft_consensus.cc:3037] T fbfb2c018e1e41df85c0c9e0d454f18f P 1854b4080cd44c4aaa0b98fd72c2da2b [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:40:46.869871 20556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbfb2c018e1e41df85c0c9e0d454f18f" candidate_uuid: "361c8462026b48e38e690a913d4d8808" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c4cc9e5cbde428e924eb7d8013ecc99"
I20260620 23:40:46.870113 20556 raft_consensus.cc:3037] T fbfb2c018e1e41df85c0c9e0d454f18f P 6c4cc9e5cbde428e924eb7d8013ecc99 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:40:46.870922 20745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:46.856270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54875 (local address 127.0.97.3:43129)
0620 23:40:46.858543 (+ 2273us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:46.858557 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:46.858579 (+ 22us) server_negotiation.cc:407] Connection header received
0620 23:40:46.858650 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:46.858658 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:46.858760 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:46.858868 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:40:46.860324 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:46.864496 (+ 4172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:46.868157 (+ 3661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:46.869625 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:46.869731 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:46.869881 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:46.870451 (+ 570us) server_negotiation.cc:299] Negotiation successful
0620 23:40:46.870590 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2142,"thread_start_us":11666,"threads_started":1}
I20260620 23:40:46.871726 20761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:46.855899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43129 (local address 127.0.97.2:54875)
0620 23:40:46.856957 (+ 1058us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:46.856983 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 23:40:46.857149 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:40:46.859558 (+ 2409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:40:46.859566 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:40:46.859607 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:40:46.860125 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:46.860140 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:46.864744 (+ 4604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:46.864753 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:46.867872 (+ 3119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:46.867909 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:46.870056 (+ 2147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:46.870062 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:46.870254 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:46.870288 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 23:40:46.871188 (+ 900us) client_negotiation.cc:240] Negotiation successful
0620 23:40:46.871355 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":861,"thread_start_us":12254,"threads_started":1}
I20260620 23:40:46.872404 20685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbfb2c018e1e41df85c0c9e0d454f18f" candidate_uuid: "6c4cc9e5cbde428e924eb7d8013ecc99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1854b4080cd44c4aaa0b98fd72c2da2b" is_pre_election: true
I20260620 23:40:46.873263 20686 raft_consensus.cc:2454] T fbfb2c018e1e41df85c0c9e0d454f18f P 1854b4080cd44c4aaa0b98fd72c2da2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 361c8462026b48e38e690a913d4d8808 in term 1.
I20260620 23:40:46.873718 20360 leader_election.cc:302] T fbfb2c018e1e41df85c0c9e0d454f18f P 361c8462026b48e38e690a913d4d8808 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1854b4080cd44c4aaa0b98fd72c2da2b, 361c8462026b48e38e690a913d4d8808; no voters:
I20260620 23:40:46.873917 20556 raft_consensus.cc:2454] T fbfb2c018e1e41df85c0c9e0d454f18f P 6c4cc9e5cbde428e924eb7d8013ecc99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 361c8462026b48e38e690a913d4d8808 in term 1.
I20260620 23:40:46.874514 20763 raft_consensus.cc:2781] T fbfb2c018e1e41df85c0c9e0d454f18f P 361c8462026b48e38e690a913d4d8808 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:40:46.874619 20755 leader_election.cc:288] T fbfb2c018e1e41df85c0c9e0d454f18f P 361c8462026b48e38e690a913d4d8808 [CANDIDATE]: Term 1 election: Requested vote from peers 1854b4080cd44c4aaa0b98fd72c2da2b (127.0.97.3:43129), 6c4cc9e5cbde428e924eb7d8013ecc99 (127.0.97.2:43295)
I20260620 23:40:46.875236 20763 raft_consensus.cc:686] T fbfb2c018e1e41df85c0c9e0d454f18f P 361c8462026b48e38e690a913d4d8808 [term 1 LEADER]: Becoming Leader. State: Replica: 361c8462026b48e38e690a913d4d8808, State: Running, Role: LEADER
I20260620 23:40:46.875937 20763 consensus_queue.cc:237] T fbfb2c018e1e41df85c0c9e0d454f18f P 361c8462026b48e38e690a913d4d8808 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1854b4080cd44c4aaa0b98fd72c2da2b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43129 } } peers { permanent_uuid: "361c8462026b48e38e690a913d4d8808" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45065 } } peers { permanent_uuid: "6c4cc9e5cbde428e924eb7d8013ecc99" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43295 } }
I20260620 23:40:46.876638 20760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:46.856161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45065 (local address 127.0.97.2:50031)
0620 23:40:46.868506 (+ 12345us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:46.868526 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:40:46.868714 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:40:46.869224 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:40:46.869231 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:40:46.869258 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:40:46.869588 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:46.869600 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:46.870978 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:46.870983 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:46.873842 (+ 2859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:40:46.873854 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:40:46.875847 (+ 1993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:40:46.875854 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:40:46.876021 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:46.876057 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 23:40:46.876167 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 23:40:46.876351 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12209,"thread_start_us":570,"threads_started":1}
I20260620 23:40:46.877295 20746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:46.856623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50031 (local address 127.0.97.1:45065)
0620 23:40:46.856787 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:46.856793 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:46.868606 (+ 11813us) server_negotiation.cc:407] Connection header received
0620 23:40:46.868867 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:46.868874 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:46.868973 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:46.869035 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:40:46.869766 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:46.870766 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:46.874043 (+ 3277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:46.875628 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:46.875767 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:46.875869 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:46.876875 (+ 1006us) server_negotiation.cc:299] Negotiation successful
0620 23:40:46.877015 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 23:40:46.878149 20425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbfb2c018e1e41df85c0c9e0d454f18f" candidate_uuid: "6c4cc9e5cbde428e924eb7d8013ecc99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "361c8462026b48e38e690a913d4d8808" is_pre_election: true
I20260620 23:40:46.879608 20490 leader_election.cc:302] T fbfb2c018e1e41df85c0c9e0d454f18f P 6c4cc9e5cbde428e924eb7d8013ecc99 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c4cc9e5cbde428e924eb7d8013ecc99; no voters: 1854b4080cd44c4aaa0b98fd72c2da2b, 361c8462026b48e38e690a913d4d8808
W20260620 23:40:46.879866 20732 tablet.cc:2369] T fbfb2c018e1e41df85c0c9e0d454f18f P 1854b4080cd44c4aaa0b98fd72c2da2b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:40:46.880352 20757 raft_consensus.cc:2726] T fbfb2c018e1e41df85c0c9e0d454f18f P 6c4cc9e5cbde428e924eb7d8013ecc99 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 23:40:46.881700 20284 catalog_manager.cc:5495] T fbfb2c018e1e41df85c0c9e0d454f18f P 361c8462026b48e38e690a913d4d8808 reported cstate change: term changed from 0 to 1, leader changed from <none> to 361c8462026b48e38e690a913d4d8808 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "361c8462026b48e38e690a913d4d8808" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1854b4080cd44c4aaa0b98fd72c2da2b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "361c8462026b48e38e690a913d4d8808" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45065 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c4cc9e5cbde428e924eb7d8013ecc99" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43295 } health_report { overall_health: UNKNOWN } } }
W20260620 23:40:46.886924 20602 tablet.cc:2369] T fbfb2c018e1e41df85c0c9e0d454f18f P 6c4cc9e5cbde428e924eb7d8013ecc99: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:40:46.921001 20471 tablet.cc:2369] T fbfb2c018e1e41df85c0c9e0d454f18f P 361c8462026b48e38e690a913d4d8808: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:40:46.934391 20746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:46.929461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53120 (local address 127.0.97.1:45065)
0620 23:40:46.929623 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:46.929628 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:46.929644 (+ 16us) server_negotiation.cc:407] Connection header received
0620 23:40:46.929783 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:46.929788 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:46.929872 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:46.929991 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:40:46.930301 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:46.931124 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:46.932470 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:46.932870 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:46.932967 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:46.933392 (+ 425us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:40:46.933900 (+ 508us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:40:46.934019 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:46.934080 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 23:40:46.934140 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 23:40:46.942737 20686 raft_consensus.cc:1261] T fbfb2c018e1e41df85c0c9e0d454f18f P 1854b4080cd44c4aaa0b98fd72c2da2b [term 1 FOLLOWER]: Refusing update from remote peer 361c8462026b48e38e690a913d4d8808: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:40:46.942799 20556 raft_consensus.cc:1261] T fbfb2c018e1e41df85c0c9e0d454f18f P 6c4cc9e5cbde428e924eb7d8013ecc99 [term 1 FOLLOWER]: Refusing update from remote peer 361c8462026b48e38e690a913d4d8808: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:40:46.943686 20763 consensus_queue.cc:1035] T fbfb2c018e1e41df85c0c9e0d454f18f P 361c8462026b48e38e690a913d4d8808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c4cc9e5cbde428e924eb7d8013ecc99" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:40:46.944011 20755 consensus_queue.cc:1035] T fbfb2c018e1e41df85c0c9e0d454f18f P 361c8462026b48e38e690a913d4d8808 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1854b4080cd44c4aaa0b98fd72c2da2b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:40:46.956682 20767 mvcc.cc:205] Tried to move back new op lower bound from 7299067277067067392 to 7299067276808933376. Current Snapshot: MvccSnapshot[applied={T|T < 7299067277067067392}]
I20260620 23:40:46.969724 20283 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:40:46.970067 20283 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:40:46.973878 20283 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a06d41d6d8f94bb69b2f279c34d98776: Sending DeleteTablet for 3 replicas of tablet fbfb2c018e1e41df85c0c9e0d454f18f
I20260620 23:40:46.975154 20666 tablet_service.cc:1514] Processing DeleteTablet for tablet fbfb2c018e1e41df85c0c9e0d454f18f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:40:46 UTC) from {username='slave'} at 127.0.0.1:53454
I20260620 23:40:46.975152 20536 tablet_service.cc:1514] Processing DeleteTablet for tablet fbfb2c018e1e41df85c0c9e0d454f18f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:40:46 UTC) from {username='slave'} at 127.0.0.1:33510
I20260620 23:40:46.975152 20405 tablet_service.cc:1514] Processing DeleteTablet for tablet fbfb2c018e1e41df85c0c9e0d454f18f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:40:46 UTC) from {username='slave'} at 127.0.0.1:53104
I20260620 23:40:46.978199 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 20343
I20260620 23:40:46.987690 20777 tablet_replica.cc:330] T fbfb2c018e1e41df85c0c9e0d454f18f P 6c4cc9e5cbde428e924eb7d8013ecc99: stopping tablet replica
I20260620 23:40:46.988312 20777 raft_consensus.cc:2229] T fbfb2c018e1e41df85c0c9e0d454f18f P 6c4cc9e5cbde428e924eb7d8013ecc99 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:40:46.988955 20777 raft_consensus.cc:2258] T fbfb2c018e1e41df85c0c9e0d454f18f P 6c4cc9e5cbde428e924eb7d8013ecc99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:40:46.989854 20776 tablet_replica.cc:330] T fbfb2c018e1e41df85c0c9e0d454f18f P 1854b4080cd44c4aaa0b98fd72c2da2b: stopping tablet replica
I20260620 23:40:46.990406 20776 raft_consensus.cc:2229] T fbfb2c018e1e41df85c0c9e0d454f18f P 1854b4080cd44c4aaa0b98fd72c2da2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:40:46.991186 20776 raft_consensus.cc:2258] T fbfb2c018e1e41df85c0c9e0d454f18f P 1854b4080cd44c4aaa0b98fd72c2da2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:40:46.994289 20776 ts_tablet_manager.cc:1905] T fbfb2c018e1e41df85c0c9e0d454f18f P 1854b4080cd44c4aaa0b98fd72c2da2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:40:47.001106 20777 ts_tablet_manager.cc:1905] T fbfb2c018e1e41df85c0c9e0d454f18f P 6c4cc9e5cbde428e924eb7d8013ecc99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:40:47.003669 20746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:46.980067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53128 (local address 127.0.97.1:45065)
0620 23:40:46.980316 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:46.980325 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:46.994499 (+ 14174us) server_negotiation.cc:407] Connection header received
0620 23:40:46.995084 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:46.995092 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:46.995242 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:46.995323 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:40:46.996053 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:46.997222 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:46.999234 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:46.999919 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:47.000020 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:47.001832 (+ 1812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:40:47.001843 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:40:47.001860 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:40:47.002682 (+ 822us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:40:47.002744 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:40:47.002771 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:40:47.002870 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:47.003182 (+ 312us) server_negotiation.cc:299] Negotiation successful
0620 23:40:47.003290 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 23:40:47.005020 20778 tablet_replica.cc:330] T fbfb2c018e1e41df85c0c9e0d454f18f P 361c8462026b48e38e690a913d4d8808: stopping tablet replica
I20260620 23:40:47.005300 20465 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:40:47.008461 20776 ts_tablet_manager.cc:1918] T fbfb2c018e1e41df85c0c9e0d454f18f P 1854b4080cd44c4aaa0b98fd72c2da2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:40:47.008661 20776 log.cc:1196] T fbfb2c018e1e41df85c0c9e0d454f18f P 1854b4080cd44c4aaa0b98fd72c2da2b: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998525501163-388-0/minicluster-data/ts-2/wal/wals/fbfb2c018e1e41df85c0c9e0d454f18f
I20260620 23:40:47.009135 20776 ts_tablet_manager.cc:1939] T fbfb2c018e1e41df85c0c9e0d454f18f P 1854b4080cd44c4aaa0b98fd72c2da2b: Deleting consensus metadata
I20260620 23:40:47.010736 20777 ts_tablet_manager.cc:1918] T fbfb2c018e1e41df85c0c9e0d454f18f P 6c4cc9e5cbde428e924eb7d8013ecc99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:40:47.011022 20777 log.cc:1196] T fbfb2c018e1e41df85c0c9e0d454f18f P 6c4cc9e5cbde428e924eb7d8013ecc99: Deleting WAL directory at /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998525501163-388-0/minicluster-data/ts-1/wal/wals/fbfb2c018e1e41df85c0c9e0d454f18f
I20260620 23:40:47.011024 20270 catalog_manager.cc:4841] TS 1854b4080cd44c4aaa0b98fd72c2da2b (127.0.97.3:43129): tablet fbfb2c018e1e41df85c0c9e0d454f18f (table test-table [id=cce57cc1123a4468afced481f51b1602]) successfully deleted
I20260620 23:40:47.011715 20777 ts_tablet_manager.cc:1939] T fbfb2c018e1e41df85c0c9e0d454f18f P 6c4cc9e5cbde428e924eb7d8013ecc99: Deleting consensus metadata
I20260620 23:40:47.013902 20270 catalog_manager.cc:4841] TS 6c4cc9e5cbde428e924eb7d8013ecc99 (127.0.97.2:43295): tablet fbfb2c018e1e41df85c0c9e0d454f18f (table test-table [id=cce57cc1123a4468afced481f51b1602]) successfully deleted
I20260620 23:40:47.005589 20778 raft_consensus.cc:2229] T fbfb2c018e1e41df85c0c9e0d454f18f P 361c8462026b48e38e690a913d4d8808 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:40:47.108143 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 20343
I20260620 23:40:47.108175 20778 raft_consensus.cc:2258] T fbfb2c018e1e41df85c0c9e0d454f18f P 361c8462026b48e38e690a913d4d8808 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:40:47.129371 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 20474
W20260620 23:40:47.129585 20269 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:45065 (error 108)
I20260620 23:40:47.131657 20747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:47.131173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45065 (local address 127.0.0.1:53136)
0620 23:40:47.131383 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:47.131510 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260620 23:40:47.131897 20269 catalog_manager.cc:4584] TS 361c8462026b48e38e690a913d4d8808 (127.0.97.1:45065): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fbfb2c018e1e41df85c0c9e0d454f18f: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45065: connect: Connection refused (error 111)
I20260620 23:40:47.137544 20748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:47.130321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33516 (local address 127.0.97.2:43295)
0620 23:40:47.130520 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:47.130541 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:47.130630 (+ 89us) server_negotiation.cc:407] Connection header received
0620 23:40:47.130825 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:47.130830 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:47.130976 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:47.131103 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:40:47.131555 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:47.132537 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:47.133919 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:47.134620 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:47.134757 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:47.135901 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:40:47.135916 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:40:47.135940 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:40:47.136679 (+ 739us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:40:47.136739 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:40:47.136764 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:40:47.136887 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:47.137043 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 23:40:47.137154 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 23:40:47.138379 20596 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:40:47.185406 20747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:47.184928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45065 (local address 127.0.0.1:53144)
0620 23:40:47.185150 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:47.185236 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260620 23:40:47.186407 20747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:47.186069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45065 (local address 127.0.0.1:53148)
0620 23:40:47.186215 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:47.186298 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 23:40:47.231442 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 20474
I20260620 23:40:47.247304 20747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:47.246755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45065 (local address 127.0.0.1:53154)
0620 23:40:47.247035 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:47.247128 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260620 23:40:47.248548 20747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:47.248165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45065 (local address 127.0.0.1:53160)
0620 23:40:47.248333 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:47.248414 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 23:40:47.249714 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 20604
I20260620 23:40:47.256701 20745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:47.250462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53456 (local address 127.0.97.3:43129)
0620 23:40:47.250651 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:47.250660 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:47.250681 (+ 21us) server_negotiation.cc:407] Connection header received
0620 23:40:47.250906 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:47.250911 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:47.251045 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:47.251150 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:40:47.251563 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:47.252248 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:47.253520 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:47.253946 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:47.254108 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:47.255345 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:40:47.255355 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:40:47.255370 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:40:47.255964 (+ 594us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:40:47.256013 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:40:47.256034 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:40:47.256156 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:47.256303 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 23:40:47.256412 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 23:40:47.257364 20726 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:40:47.329196 20747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:47.328710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45065 (local address 127.0.0.1:53162)
0620 23:40:47.328932 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:47.329019 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260620 23:40:47.330314 20747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:47.329934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45065 (local address 127.0.0.1:53170)
0620 23:40:47.330132 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 23:40:47.330210 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260620 23:40:47.347180 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 20604
I20260620 23:40:47.364434 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 20253
I20260620 23:40:47.365628 20314 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:40:47.476330 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 20253
2026-06-20T23:40:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2936 ms)
[----------] 6 tests from EncryptionPolicyTest (21942 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781998525501163-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:40:47 dist-test-slave-tcgx krb5kdc[20787](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:40:47 dist-test-slave-tcgx krb5kdc[20787](info): set up 2 sockets
Jun 20 23:40:47 dist-test-slave-tcgx krb5kdc[20787](info): commencing operation
krb5kdc: starting...
W20260620 23:40:48.596124 388 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781998525501163-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781998525501163-388-0/krb5kdc/test-user.keytab.
Jun 20 23:40:48 dist-test-slave-tcgx krb5kdc[20787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:40:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:40:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781998525501163-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781998525501163-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:40:48.864266 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781998525501163-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781998525501163-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781998525501163-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41151
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781998525501163-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41151
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:40:49.202632 20800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:40:49.202965 20800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:40:49.203073 20800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:40:49.210341 20800 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:40:49.210470 20800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:40:49.210520 20800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:40:49.210575 20800 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:40:49.210621 20800 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:40:49.221469 20800 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781998525501163-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781998525501163-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41151
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781998525501163-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781998525501163-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:40:49.222792 20800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:40:49.224184 20800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:40:49.236088 20806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:40:49.236122 20808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:40:49.236027 20805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:40:49.236856 20800 server_base.cc:1034] running on GCE node
I20260620 23:40:49.237680 20800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:40:49.239264 20800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:40:49.240494 20800 hybrid_clock.cc:648] HybridClock initialized: now 1781998849240438 us; error 54 us; skew 500 ppm
I20260620 23:40:49.240865 20800 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:40:49 dist-test-slave-tcgx krb5kdc[20787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998849, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:40:49.244028 20800 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:40:49.245627 20800 webserver.cc:460] Webserver started at http://127.0.0.1:37719/ using document root <none> and password file <none>
I20260620 23:40:49.246364 20800 fs_manager.cc:359] Metadata directory not provided
I20260620 23:40:49.246513 20800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:40:49.246836 20800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:40:49.249974 20800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781998525501163-388-0/minicluster-data/master-0/data/instance:
uuid: "fe8b645f6ef74fe18a246e01ddbacd74"
format_stamp: "Formatted at 2026-06-20 23:40:49 on dist-test-slave-tcgx"
server_key: "196e62d06978a6e2b68dc7c7bf3626a7"
server_key_iv: "2fe6173261b292c259a53d6753f6b186"
server_key_version: "encryptionkey@0"
I20260620 23:40:49.250780 20800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781998525501163-388-0/minicluster-data/master-0/wal/instance:
uuid: "fe8b645f6ef74fe18a246e01ddbacd74"
format_stamp: "Formatted at 2026-06-20 23:40:49 on dist-test-slave-tcgx"
server_key: "196e62d06978a6e2b68dc7c7bf3626a7"
server_key_iv: "2fe6173261b292c259a53d6753f6b186"
server_key_version: "encryptionkey@0"
I20260620 23:40:49.256199 20800 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 23:40:49.259783 20815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:49.261147 20800 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:40:49.261337 20800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781998525501163-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781998525501163-388-0/minicluster-data/master-0/wal
uuid: "fe8b645f6ef74fe18a246e01ddbacd74"
format_stamp: "Formatted at 2026-06-20 23:40:49 on dist-test-slave-tcgx"
server_key: "196e62d06978a6e2b68dc7c7bf3626a7"
server_key_iv: "2fe6173261b292c259a53d6753f6b186"
server_key_version: "encryptionkey@0"
I20260620 23:40:49.261546 20800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781998525501163-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781998525501163-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781998525501163-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:40:49.279471 20800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:40:49.283553 20800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:40:49.283866 20800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:40:49.303197 20800 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41151
I20260620 23:40:49.303234 20866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41151 every 8 connection(s)
I20260620 23:40:49.304632 20800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:40:49.308552 20867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:40:49.309871 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 20800
I20260620 23:40:49.310104 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781998525501163-388-0/minicluster-data/master-0/wal/instance
I20260620 23:40:49.310720 388 external_mini_cluster.cc:1428] Setting key 334448fa43528cc89ca7eded951c0c8d
I20260620 23:40:49.322216 20867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe8b645f6ef74fe18a246e01ddbacd74: Bootstrap starting.
Jun 20 23:40:49 dist-test-slave-tcgx krb5kdc[20787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 23:40:49.325951 20867 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe8b645f6ef74fe18a246e01ddbacd74: Neither blocks nor log segments found. Creating new log.
I20260620 23:40:49.327174 20867 log.cc:824] T 00000000000000000000000000000000 P fe8b645f6ef74fe18a246e01ddbacd74: Log is configured to *not* fsync() on all Append() calls
I20260620 23:40:49.330667 20867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe8b645f6ef74fe18a246e01ddbacd74: No bootstrap required, opened a new log
I20260620 23:40:49.334806 20870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:49.312863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56239 (local address 127.0.0.1:41151)
0620 23:40:49.313708 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:49.313721 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:49.313776 (+ 55us) server_negotiation.cc:407] Connection header received
0620 23:40:49.315532 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:49.315565 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:49.316029 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:49.316465 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:40:49.318020 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:49.319545 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:49.321036 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:49.321761 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:49.321984 (+ 223us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:49.326487 (+ 4503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:40:49.326495 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:40:49.326512 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:40:49.326562 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:40:49.329797 (+ 3235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:40:49.330870 (+ 1073us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:40:49.330879 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:40:49.330895 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:40:49.331139 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:40:49.331688 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:40:49.331691 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:40:49.331693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:40:49.332245 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:40:49.332474 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:49.333077 (+ 603us) server_negotiation.cc:299] Negotiation successful
0620 23:40:49.333471 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":385,"threads_started":1}
I20260620 23:40:49.339259 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:40:49.339762 20867 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe8b645f6ef74fe18a246e01ddbacd74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe8b645f6ef74fe18a246e01ddbacd74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41151 } }
I20260620 23:40:49.340222 20867 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe8b645f6ef74fe18a246e01ddbacd74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:40:49.340318 20867 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe8b645f6ef74fe18a246e01ddbacd74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe8b645f6ef74fe18a246e01ddbacd74, State: Initialized, Role: FOLLOWER
I20260620 23:40:49.340956 20867 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe8b645f6ef74fe18a246e01ddbacd74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe8b645f6ef74fe18a246e01ddbacd74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41151 } }
I20260620 23:40:49.341223 20867 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe8b645f6ef74fe18a246e01ddbacd74 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:40:49.341349 20867 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe8b645f6ef74fe18a246e01ddbacd74 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:40:49.341516 20867 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe8b645f6ef74fe18a246e01ddbacd74 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:40:49.345052 20867 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe8b645f6ef74fe18a246e01ddbacd74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe8b645f6ef74fe18a246e01ddbacd74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41151 } }
I20260620 23:40:49.345642 20867 leader_election.cc:302] T 00000000000000000000000000000000 P fe8b645f6ef74fe18a246e01ddbacd74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe8b645f6ef74fe18a246e01ddbacd74; no voters:
I20260620 23:40:49.346379 20867 leader_election.cc:288] T 00000000000000000000000000000000 P fe8b645f6ef74fe18a246e01ddbacd74 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:40:49.346521 20877 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe8b645f6ef74fe18a246e01ddbacd74 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:40:49.348650 20867 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe8b645f6ef74fe18a246e01ddbacd74 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:40:49.348655 20877 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe8b645f6ef74fe18a246e01ddbacd74 [term 1 LEADER]: Becoming Leader. State: Replica: fe8b645f6ef74fe18a246e01ddbacd74, State: Running, Role: LEADER
I20260620 23:40:49.349315 20877 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe8b645f6ef74fe18a246e01ddbacd74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe8b645f6ef74fe18a246e01ddbacd74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41151 } }
I20260620 23:40:49.353722 20878 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe8b645f6ef74fe18a246e01ddbacd74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe8b645f6ef74fe18a246e01ddbacd74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe8b645f6ef74fe18a246e01ddbacd74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41151 } } }
I20260620 23:40:49.353713 20879 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe8b645f6ef74fe18a246e01ddbacd74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe8b645f6ef74fe18a246e01ddbacd74. Latest consensus state: current_term: 1 leader_uuid: "fe8b645f6ef74fe18a246e01ddbacd74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe8b645f6ef74fe18a246e01ddbacd74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41151 } } }
I20260620 23:40:49.354166 20878 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe8b645f6ef74fe18a246e01ddbacd74 [sys.catalog]: This master's current role is: LEADER
I20260620 23:40:49.354166 20879 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe8b645f6ef74fe18a246e01ddbacd74 [sys.catalog]: This master's current role is: LEADER
I20260620 23:40:49.364079 20887 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:40:49.366542 20870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:49.344943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44739 (local address 127.0.0.1:41151)
0620 23:40:49.345146 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:49.345153 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:49.345709 (+ 556us) server_negotiation.cc:407] Connection header received
0620 23:40:49.345878 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:49.345883 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:49.345963 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:49.346082 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:40:49.348363 (+ 2281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:49.349611 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:49.351631 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:49.352121 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:49.352270 (+ 149us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:49.357113 (+ 4843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:40:49.357123 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:40:49.357131 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:40:49.357192 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:40:49.362494 (+ 5302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:40:49.363497 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:40:49.363504 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:40:49.363508 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:40:49.363674 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:40:49.364337 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:40:49.364346 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:40:49.364352 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:40:49.364908 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:40:49.365272 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:49.365728 (+ 456us) server_negotiation.cc:299] Negotiation successful
0620 23:40:49.366008 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 23:40:49.371706 20887 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:40:49.382090 20887 catalog_manager.cc:1347] Generated new cluster ID: 9b3fefb403ff4a26a9dc619f2f06999a
I20260620 23:40:49.382230 20887 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:40:49.401917 20887 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:40:49.403944 20887 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:40:49.423930 20887 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe8b645f6ef74fe18a246e01ddbacd74: Generated new TSK 0
I20260620 23:40:49.425115 20887 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:40:49.431356 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 20800
I20260620 23:40:49.433048 20862 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:40:49.530998 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 20800
W20260620 23:40:49.548766 20785 connection.cc:535] client connection to 127.0.0.1:41151 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41151): Connection reset by peer (error 104)
2026-06-20T23:40:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2047 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781998525501163-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:40:49 dist-test-slave-tcgx krb5kdc[20899](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:40:49 dist-test-slave-tcgx krb5kdc[20899](info): set up 2 sockets
Jun 20 23:40:49 dist-test-slave-tcgx krb5kdc[20899](info): commencing operation
krb5kdc: starting...
W20260620 23:40:50.643429 388 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781998525501163-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781998525501163-388-0/krb5kdc/test-user.keytab.
Jun 20 23:40:50 dist-test-slave-tcgx krb5kdc[20899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:40:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:40:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781998525501163-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781998525501163-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:40:50.910090 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781998525501163-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781998525501163-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781998525501163-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40457
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781998525501163-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40457
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:40:51.251317 20912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:40:51.251641 20912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:40:51.251835 20912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:40:51.259433 20912 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:40:51.259562 20912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:40:51.259678 20912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:40:51.259784 20912 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:40:51.259856 20912 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:40:51.270279 20912 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781998525501163-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781998525501163-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40457
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781998525501163-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781998525501163-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:40:51.271760 20912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:40:51.273105 20912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:40:51.285816 20920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:40:51.285822 20918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:40:51.285858 20917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:40:51.286738 20912 server_base.cc:1034] running on GCE node
I20260620 23:40:51.287636 20912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:40:51.289203 20912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:40:51.290467 20912 hybrid_clock.cc:648] HybridClock initialized: now 1781998851290408 us; error 39 us; skew 500 ppm
I20260620 23:40:51.290845 20912 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:40:51 dist-test-slave-tcgx krb5kdc[20899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998851, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:40:51.299592 20912 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:40:51.301316 20912 webserver.cc:460] Webserver started at http://127.0.0.1:35829/ using document root <none> and password file <none>
I20260620 23:40:51.302073 20912 fs_manager.cc:359] Metadata directory not provided
I20260620 23:40:51.302222 20912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:40:51.302556 20912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:40:51.305732 20912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781998525501163-388-0/minicluster-data/master-0/data/instance:
uuid: "2060d63690dd40fbb4de0d7cc67283bf"
format_stamp: "Formatted at 2026-06-20 23:40:51 on dist-test-slave-tcgx"
server_key: "0517b08c92c7fbf1418d38c726532492"
server_key_iv: "02f9b129ab5b586b7cd62854197d2616"
server_key_version: "encryptionkey@0"
I20260620 23:40:51.306521 20912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781998525501163-388-0/minicluster-data/master-0/wal/instance:
uuid: "2060d63690dd40fbb4de0d7cc67283bf"
format_stamp: "Formatted at 2026-06-20 23:40:51 on dist-test-slave-tcgx"
server_key: "0517b08c92c7fbf1418d38c726532492"
server_key_iv: "02f9b129ab5b586b7cd62854197d2616"
server_key_version: "encryptionkey@0"
I20260620 23:40:51.312119 20912 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 23:40:51.315930 20927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:51.317579 20912 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:40:51.317793 20912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781998525501163-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781998525501163-388-0/minicluster-data/master-0/wal
uuid: "2060d63690dd40fbb4de0d7cc67283bf"
format_stamp: "Formatted at 2026-06-20 23:40:51 on dist-test-slave-tcgx"
server_key: "0517b08c92c7fbf1418d38c726532492"
server_key_iv: "02f9b129ab5b586b7cd62854197d2616"
server_key_version: "encryptionkey@0"
I20260620 23:40:51.318032 20912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781998525501163-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781998525501163-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781998525501163-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:40:51.348636 20912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:40:51.352835 20912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:40:51.353149 20912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:40:51.372243 20912 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40457
I20260620 23:40:51.372272 20978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40457 every 8 connection(s)
I20260620 23:40:51.373723 20912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:40:51.375963 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 20912
I20260620 23:40:51.376168 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781998525501163-388-0/minicluster-data/master-0/wal/instance
I20260620 23:40:51.376698 388 external_mini_cluster.cc:1428] Setting key 2f3d9aa6b8edd1db6ba712ed0c790eb8
I20260620 23:40:51.378522 20979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:40:51.391119 20982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:51.378563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49759 (local address 127.0.0.1:40457)
0620 23:40:51.379477 (+ 914us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:51.379489 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:51.379532 (+ 43us) server_negotiation.cc:407] Connection header received
0620 23:40:51.381198 (+ 1666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:51.381220 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:51.381695 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:51.382085 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:40:51.382700 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:51.384159 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:51.386077 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:51.386795 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:51.387013 (+ 218us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:51.388398 (+ 1385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:40:51.388406 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:40:51.388425 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:40:51.389061 (+ 636us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:40:51.389148 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:40:51.389176 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:40:51.389320 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:51.389712 (+ 392us) server_negotiation.cc:299] Negotiation successful
0620 23:40:51.389998 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":417,"threads_started":1}
I20260620 23:40:51.392138 20979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2060d63690dd40fbb4de0d7cc67283bf: Bootstrap starting.
I20260620 23:40:51.395398 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:40:51.395632 20979 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2060d63690dd40fbb4de0d7cc67283bf: Neither blocks nor log segments found. Creating new log.
I20260620 23:40:51.396883 20979 log.cc:824] T 00000000000000000000000000000000 P 2060d63690dd40fbb4de0d7cc67283bf: Log is configured to *not* fsync() on all Append() calls
I20260620 23:40:51.400542 20979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2060d63690dd40fbb4de0d7cc67283bf: No bootstrap required, opened a new log
I20260620 23:40:51.407811 20982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:51.400754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42291 (local address 127.0.0.1:40457)
0620 23:40:51.400943 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:51.400950 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:51.401403 (+ 453us) server_negotiation.cc:407] Connection header received
0620 23:40:51.401575 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:51.401580 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:51.401654 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:51.401769 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:40:51.402266 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:51.403525 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:51.404977 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:51.405686 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:51.405831 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:51.406982 (+ 1151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:40:51.406989 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:40:51.406993 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:40:51.407015 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:40:51.407077 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:40:51.407099 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:40:51.407220 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:51.407396 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 23:40:51.407455 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 23:40:51.410310 20979 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2060d63690dd40fbb4de0d7cc67283bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2060d63690dd40fbb4de0d7cc67283bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40457 } }
I20260620 23:40:51.410687 20979 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2060d63690dd40fbb4de0d7cc67283bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:40:51.410761 20979 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2060d63690dd40fbb4de0d7cc67283bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2060d63690dd40fbb4de0d7cc67283bf, State: Initialized, Role: FOLLOWER
I20260620 23:40:51.411361 20979 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2060d63690dd40fbb4de0d7cc67283bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2060d63690dd40fbb4de0d7cc67283bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40457 } }
I20260620 23:40:51.411566 20979 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2060d63690dd40fbb4de0d7cc67283bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:40:51.411670 20979 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2060d63690dd40fbb4de0d7cc67283bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:40:51.411798 20979 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2060d63690dd40fbb4de0d7cc67283bf [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:40:51.413942 20979 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2060d63690dd40fbb4de0d7cc67283bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2060d63690dd40fbb4de0d7cc67283bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40457 } }
I20260620 23:40:51.414408 20979 leader_election.cc:302] T 00000000000000000000000000000000 P 2060d63690dd40fbb4de0d7cc67283bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2060d63690dd40fbb4de0d7cc67283bf; no voters:
I20260620 23:40:51.415009 20979 leader_election.cc:288] T 00000000000000000000000000000000 P 2060d63690dd40fbb4de0d7cc67283bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:40:51.415184 20989 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2060d63690dd40fbb4de0d7cc67283bf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:40:51.415977 20989 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2060d63690dd40fbb4de0d7cc67283bf [term 1 LEADER]: Becoming Leader. State: Replica: 2060d63690dd40fbb4de0d7cc67283bf, State: Running, Role: LEADER
I20260620 23:40:51.416513 20989 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2060d63690dd40fbb4de0d7cc67283bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2060d63690dd40fbb4de0d7cc67283bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40457 } }
I20260620 23:40:51.417186 20979 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2060d63690dd40fbb4de0d7cc67283bf [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:40:51.419603 20991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2060d63690dd40fbb4de0d7cc67283bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2060d63690dd40fbb4de0d7cc67283bf. Latest consensus state: current_term: 1 leader_uuid: "2060d63690dd40fbb4de0d7cc67283bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2060d63690dd40fbb4de0d7cc67283bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40457 } } }
I20260620 23:40:51.420046 20991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2060d63690dd40fbb4de0d7cc67283bf [sys.catalog]: This master's current role is: LEADER
I20260620 23:40:51.421670 20990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2060d63690dd40fbb4de0d7cc67283bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2060d63690dd40fbb4de0d7cc67283bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2060d63690dd40fbb4de0d7cc67283bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40457 } } }
I20260620 23:40:51.421967 20990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2060d63690dd40fbb4de0d7cc67283bf [sys.catalog]: This master's current role is: LEADER
I20260620 23:40:51.422398 20998 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:40:51.428431 20998 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:40:51.437898 20998 catalog_manager.cc:1347] Generated new cluster ID: c5a748b0c8ca43998753b4934a40f700
I20260620 23:40:51.438042 20998 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:40:51.472154 20998 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:40:51.473508 20998 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:40:51.491057 20998 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2060d63690dd40fbb4de0d7cc67283bf: Generated new TSK 0
I20260620 23:40:51.492172 20998 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:40:51.515148 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 20912
W20260620 23:40:51.518574 20974 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49759
W20260620 23:40:51.519608 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 20912: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:40:51.519835 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 20912
W20260620 23:40:51.537163 20897 connection.cc:535] client connection to 127.0.0.1:40457 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40457): Connection reset by peer (error 104)
2026-06-20T23:40:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1988 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781998525501163-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:40:51 dist-test-slave-tcgx krb5kdc[21010](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:40:51 dist-test-slave-tcgx krb5kdc[21010](info): set up 2 sockets
Jun 20 23:40:51 dist-test-slave-tcgx krb5kdc[21010](info): commencing operation
krb5kdc: starting...
W20260620 23:40:52.631395 388 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.006s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781998525501163-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781998525501163-388-0/krb5kdc/test-user.keytab.
Jun 20 23:40:52 dist-test-slave-tcgx krb5kdc[21010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:40:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:40:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781998525501163-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781998525501163-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:40:52.894336 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781998525501163-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781998525501163-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781998525501163-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44427
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781998525501163-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44427
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:40:53.234549 21023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:40:53.234864 21023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:40:53.235049 21023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:40:53.242447 21023 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:40:53.242571 21023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:40:53.242619 21023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:40:53.242681 21023 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:40:53.242727 21023 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:40:53.253211 21023 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781998525501163-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781998525501163-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44427
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781998525501163-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781998525501163-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:40:53.254534 21023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:40:53.255936 21023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:40:53.268606 21028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:40:53.268616 21029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:40:53.268673 21031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:40:53.269366 21023 server_base.cc:1034] running on GCE node
I20260620 23:40:53.270226 21023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:40:53.271855 21023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:40:53.273108 21023 hybrid_clock.cc:648] HybridClock initialized: now 1781998853273035 us; error 58 us; skew 500 ppm
I20260620 23:40:53.273517 21023 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:40:53 dist-test-slave-tcgx krb5kdc[21010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998853, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:40:53.276794 21023 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:40:53.278342 21023 webserver.cc:460] Webserver started at http://127.0.0.1:44411/ using document root <none> and password file <none>
I20260620 23:40:53.279124 21023 fs_manager.cc:359] Metadata directory not provided
I20260620 23:40:53.279259 21023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:40:53.279582 21023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:40:53.282644 21023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781998525501163-388-0/minicluster-data/master-0/data/instance:
uuid: "03bde18481e54eaa8241ef137df32466"
format_stamp: "Formatted at 2026-06-20 23:40:53 on dist-test-slave-tcgx"
server_key: "2e5fbfc0d638b82276c111239cb6ce13"
server_key_iv: "4884d9e0ea9bd204e150ca4ca85a3760"
server_key_version: "encryptionkey@0"
I20260620 23:40:53.283537 21023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781998525501163-388-0/minicluster-data/master-0/wal/instance:
uuid: "03bde18481e54eaa8241ef137df32466"
format_stamp: "Formatted at 2026-06-20 23:40:53 on dist-test-slave-tcgx"
server_key: "2e5fbfc0d638b82276c111239cb6ce13"
server_key_iv: "4884d9e0ea9bd204e150ca4ca85a3760"
server_key_version: "encryptionkey@0"
I20260620 23:40:53.288837 21023 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:40:53.292493 21038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:53.293995 21023 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:40:53.294256 21023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781998525501163-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781998525501163-388-0/minicluster-data/master-0/wal
uuid: "03bde18481e54eaa8241ef137df32466"
format_stamp: "Formatted at 2026-06-20 23:40:53 on dist-test-slave-tcgx"
server_key: "2e5fbfc0d638b82276c111239cb6ce13"
server_key_iv: "4884d9e0ea9bd204e150ca4ca85a3760"
server_key_version: "encryptionkey@0"
I20260620 23:40:53.294474 21023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781998525501163-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781998525501163-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781998525501163-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:40:53.300352 21023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:40:53.304618 21023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:40:53.304922 21023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:40:53.324358 21023 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44427
I20260620 23:40:53.324396 21089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44427 every 8 connection(s)
I20260620 23:40:53.325863 21023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:40:53.329339 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 21023
I20260620 23:40:53.329546 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781998525501163-388-0/minicluster-data/master-0/wal/instance
I20260620 23:40:53.329988 388 external_mini_cluster.cc:1428] Setting key 047595eafc1292085ceb3b09b69ce439
I20260620 23:40:53.330029 21090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:40:53.337913 21093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:53.331669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48835 (local address 127.0.0.1:44427)
0620 23:40:53.332404 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:53.332417 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:53.332461 (+ 44us) server_negotiation.cc:407] Connection header received
0620 23:40:53.334137 (+ 1676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:53.334162 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:53.334570 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:53.335034 (+ 464us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:40:53.336155 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:40:53.336161 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:40:53.336178 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:40:53.336195 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:40:53.336268 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:40:53.336288 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:40:53.336553 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:53.336702 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 23:40:53.336907 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":297,"threads_started":1}
I20260620 23:40:53.342257 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:40:53.344606 21090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03bde18481e54eaa8241ef137df32466: Bootstrap starting.
I20260620 23:40:53.348572 21090 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03bde18481e54eaa8241ef137df32466: Neither blocks nor log segments found. Creating new log.
I20260620 23:40:53.349452 21093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:53.346803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40973 (local address 127.0.0.1:44427)
0620 23:40:53.347039 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:53.347047 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:53.347354 (+ 307us) server_negotiation.cc:407] Connection header received
0620 23:40:53.347558 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:53.347561 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:53.347612 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:53.347694 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:40:53.348775 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:40:53.348779 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:40:53.348782 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:40:53.348798 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:40:53.348839 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:40:53.348854 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:40:53.348947 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:53.349110 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 23:40:53.349163 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 23:40:53.350028 21090 log.cc:824] T 00000000000000000000000000000000 P 03bde18481e54eaa8241ef137df32466: Log is configured to *not* fsync() on all Append() calls
I20260620 23:40:53.353147 21090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03bde18481e54eaa8241ef137df32466: No bootstrap required, opened a new log
I20260620 23:40:53.361183 21090 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03bde18481e54eaa8241ef137df32466 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03bde18481e54eaa8241ef137df32466" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44427 } }
I20260620 23:40:53.361578 21090 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03bde18481e54eaa8241ef137df32466 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:40:53.361653 21090 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03bde18481e54eaa8241ef137df32466 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03bde18481e54eaa8241ef137df32466, State: Initialized, Role: FOLLOWER
I20260620 23:40:53.362331 21090 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03bde18481e54eaa8241ef137df32466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03bde18481e54eaa8241ef137df32466" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44427 } }
I20260620 23:40:53.362671 21090 raft_consensus.cc:388] T 00000000000000000000000000000000 P 03bde18481e54eaa8241ef137df32466 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:40:53.362855 21090 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03bde18481e54eaa8241ef137df32466 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:40:53.363065 21090 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03bde18481e54eaa8241ef137df32466 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:40:53.365321 21090 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03bde18481e54eaa8241ef137df32466 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03bde18481e54eaa8241ef137df32466" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44427 } }
I20260620 23:40:53.365900 21090 leader_election.cc:302] T 00000000000000000000000000000000 P 03bde18481e54eaa8241ef137df32466 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03bde18481e54eaa8241ef137df32466; no voters:
I20260620 23:40:53.366515 21090 leader_election.cc:288] T 00000000000000000000000000000000 P 03bde18481e54eaa8241ef137df32466 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:40:53.366685 21100 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03bde18481e54eaa8241ef137df32466 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:40:53.367606 21100 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03bde18481e54eaa8241ef137df32466 [term 1 LEADER]: Becoming Leader. State: Replica: 03bde18481e54eaa8241ef137df32466, State: Running, Role: LEADER
I20260620 23:40:53.368243 21100 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03bde18481e54eaa8241ef137df32466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03bde18481e54eaa8241ef137df32466" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44427 } }
I20260620 23:40:53.369081 21090 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03bde18481e54eaa8241ef137df32466 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:40:53.371381 21101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03bde18481e54eaa8241ef137df32466 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "03bde18481e54eaa8241ef137df32466" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03bde18481e54eaa8241ef137df32466" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44427 } } }
I20260620 23:40:53.371711 21102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03bde18481e54eaa8241ef137df32466 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03bde18481e54eaa8241ef137df32466. Latest consensus state: current_term: 1 leader_uuid: "03bde18481e54eaa8241ef137df32466" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03bde18481e54eaa8241ef137df32466" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44427 } } }
I20260620 23:40:53.371912 21101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03bde18481e54eaa8241ef137df32466 [sys.catalog]: This master's current role is: LEADER
I20260620 23:40:53.371991 21102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03bde18481e54eaa8241ef137df32466 [sys.catalog]: This master's current role is: LEADER
I20260620 23:40:53.374563 21108 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:40:53.379609 21108 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:40:53.388670 21108 catalog_manager.cc:1347] Generated new cluster ID: 568c90e27c4945a5850b9d0745ade6ba
I20260620 23:40:53.388806 21108 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:40:53.427963 21108 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:40:53.428504 21108 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:40:53.442788 21108 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03bde18481e54eaa8241ef137df32466: Generated new TSK 0
I20260620 23:40:53.443970 21108 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:40:53.453996 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 21023
W20260620 23:40:53.455343 21085 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:48835
W20260620 23:40:53.456120 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 21023: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:40:53.456279 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 21023
2026-06-20T23:40:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1936 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781998525501163-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:40:53 dist-test-slave-tcgx krb5kdc[21121](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:40:53 dist-test-slave-tcgx krb5kdc[21121](info): set up 2 sockets
Jun 20 23:40:53 dist-test-slave-tcgx krb5kdc[21121](info): commencing operation
krb5kdc: starting...
W20260620 23:40:54.575388 388 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781998525501163-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781998525501163-388-0/krb5kdc/test-user.keytab.
Jun 20 23:40:54 dist-test-slave-tcgx krb5kdc[21121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:40:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:40:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781998525501163-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781998525501163-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:40:54.838254 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781998525501163-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781998525501163-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781998525501163-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34969
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781998525501163-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34969
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:40:55.176942 21134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:40:55.177244 21134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:40:55.177413 21134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:40:55.184738 21134 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:40:55.184860 21134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:40:55.184953 21134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:40:55.185045 21134 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:40:55.185115 21134 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:40:55.195367 21134 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781998525501163-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781998525501163-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781998525501163-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781998525501163-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:40:55.196733 21134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:40:55.198486 21134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:40:55.210501 21140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:40:55.210583 21139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:40:55.210511 21142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:40:55.211437 21134 server_base.cc:1034] running on GCE node
I20260620 23:40:55.212268 21134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:40:55.213758 21134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:40:55.215039 21134 hybrid_clock.cc:648] HybridClock initialized: now 1781998855214955 us; error 54 us; skew 500 ppm
I20260620 23:40:55.215408 21134 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:40:55 dist-test-slave-tcgx krb5kdc[21121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998855, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:40:55.218655 21134 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:40:55.220463 21134 webserver.cc:460] Webserver started at http://127.0.0.1:32951/ using document root <none> and password file <none>
I20260620 23:40:55.221186 21134 fs_manager.cc:359] Metadata directory not provided
I20260620 23:40:55.221331 21134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:40:55.221657 21134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:40:55.224726 21134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781998525501163-388-0/minicluster-data/master-0/data/instance:
uuid: "3a611e9f48f24ec1bdb3f8d0ec13ffc1"
format_stamp: "Formatted at 2026-06-20 23:40:55 on dist-test-slave-tcgx"
server_key: "01a0f0bb82c7cba9e7083eb39d6e0b97"
server_key_iv: "b446c94b2d344d32fef62fab7efaf861"
server_key_version: "encryptionkey@0"
I20260620 23:40:55.225510 21134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781998525501163-388-0/minicluster-data/master-0/wal/instance:
uuid: "3a611e9f48f24ec1bdb3f8d0ec13ffc1"
format_stamp: "Formatted at 2026-06-20 23:40:55 on dist-test-slave-tcgx"
server_key: "01a0f0bb82c7cba9e7083eb39d6e0b97"
server_key_iv: "b446c94b2d344d32fef62fab7efaf861"
server_key_version: "encryptionkey@0"
I20260620 23:40:55.230890 21134 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 23:40:55.234457 21149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:55.235934 21134 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:40:55.236137 21134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781998525501163-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781998525501163-388-0/minicluster-data/master-0/wal
uuid: "3a611e9f48f24ec1bdb3f8d0ec13ffc1"
format_stamp: "Formatted at 2026-06-20 23:40:55 on dist-test-slave-tcgx"
server_key: "01a0f0bb82c7cba9e7083eb39d6e0b97"
server_key_iv: "b446c94b2d344d32fef62fab7efaf861"
server_key_version: "encryptionkey@0"
I20260620 23:40:55.236354 21134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781998525501163-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781998525501163-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781998525501163-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:40:55.250560 21134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:40:55.254889 21134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:40:55.255244 21134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:40:55.274354 21134 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34969
I20260620 23:40:55.274384 21200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34969 every 8 connection(s)
I20260620 23:40:55.275887 21134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:40:55.279960 21201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:40:55.284168 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 21134
I20260620 23:40:55.284366 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781998525501163-388-0/minicluster-data/master-0/wal/instance
I20260620 23:40:55.284919 388 external_mini_cluster.cc:1428] Setting key 2b8ada91a8ede183cd221499b74421bd
I20260620 23:40:55.292054 21201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a611e9f48f24ec1bdb3f8d0ec13ffc1: Bootstrap starting.
I20260620 23:40:55.296008 21201 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a611e9f48f24ec1bdb3f8d0ec13ffc1: Neither blocks nor log segments found. Creating new log.
I20260620 23:40:55.297204 21201 log.cc:824] T 00000000000000000000000000000000 P 3a611e9f48f24ec1bdb3f8d0ec13ffc1: Log is configured to *not* fsync() on all Append() calls
I20260620 23:40:55.301069 21201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a611e9f48f24ec1bdb3f8d0ec13ffc1: No bootstrap required, opened a new log
Jun 20 23:40:55 dist-test-slave-tcgx krb5kdc[21121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 23:40:55.311769 21204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:55.287156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50369 (local address 127.0.0.1:34969)
0620 23:40:55.288035 (+ 879us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:55.288047 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:55.288087 (+ 40us) server_negotiation.cc:407] Connection header received
0620 23:40:55.290398 (+ 2311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:55.290424 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:55.290849 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:55.291254 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:40:55.293190 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:55.294569 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:55.296494 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:55.297184 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:55.297318 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:55.302340 (+ 5022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:40:55.302350 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:40:55.302370 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:40:55.302417 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:40:55.306629 (+ 4212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:40:55.307564 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:40:55.307574 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:40:55.307584 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:40:55.307824 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:40:55.308273 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:40:55.308278 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:40:55.308282 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:40:55.309109 (+ 827us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:40:55.309436 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:55.309882 (+ 446us) server_negotiation.cc:299] Negotiation successful
0620 23:40:55.310403 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":413,"threads_started":1}
I20260620 23:40:55.312427 21201 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a611e9f48f24ec1bdb3f8d0ec13ffc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a611e9f48f24ec1bdb3f8d0ec13ffc1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34969 } }
I20260620 23:40:55.312925 21201 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a611e9f48f24ec1bdb3f8d0ec13ffc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:40:55.313043 21201 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a611e9f48f24ec1bdb3f8d0ec13ffc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a611e9f48f24ec1bdb3f8d0ec13ffc1, State: Initialized, Role: FOLLOWER
I20260620 23:40:55.313705 21201 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a611e9f48f24ec1bdb3f8d0ec13ffc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a611e9f48f24ec1bdb3f8d0ec13ffc1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34969 } }
I20260620 23:40:55.313962 21201 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a611e9f48f24ec1bdb3f8d0ec13ffc1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:40:55.314129 21201 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a611e9f48f24ec1bdb3f8d0ec13ffc1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:40:55.314306 21201 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a611e9f48f24ec1bdb3f8d0ec13ffc1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:40:55.316133 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:40:55.317122 21201 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a611e9f48f24ec1bdb3f8d0ec13ffc1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a611e9f48f24ec1bdb3f8d0ec13ffc1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34969 } }
I20260620 23:40:55.317670 21201 leader_election.cc:302] T 00000000000000000000000000000000 P 3a611e9f48f24ec1bdb3f8d0ec13ffc1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a611e9f48f24ec1bdb3f8d0ec13ffc1; no voters:
I20260620 23:40:55.318323 21201 leader_election.cc:288] T 00000000000000000000000000000000 P 3a611e9f48f24ec1bdb3f8d0ec13ffc1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:40:55.318529 21206 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a611e9f48f24ec1bdb3f8d0ec13ffc1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:40:55.320134 21206 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a611e9f48f24ec1bdb3f8d0ec13ffc1 [term 1 LEADER]: Becoming Leader. State: Replica: 3a611e9f48f24ec1bdb3f8d0ec13ffc1, State: Running, Role: LEADER
I20260620 23:40:55.320788 21201 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a611e9f48f24ec1bdb3f8d0ec13ffc1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:40:55.321023 21206 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a611e9f48f24ec1bdb3f8d0ec13ffc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a611e9f48f24ec1bdb3f8d0ec13ffc1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34969 } }
I20260620 23:40:55.324347 21210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a611e9f48f24ec1bdb3f8d0ec13ffc1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a611e9f48f24ec1bdb3f8d0ec13ffc1. Latest consensus state: current_term: 1 leader_uuid: "3a611e9f48f24ec1bdb3f8d0ec13ffc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a611e9f48f24ec1bdb3f8d0ec13ffc1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34969 } } }
I20260620 23:40:55.324777 21210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a611e9f48f24ec1bdb3f8d0ec13ffc1 [sys.catalog]: This master's current role is: LEADER
I20260620 23:40:55.325366 21207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a611e9f48f24ec1bdb3f8d0ec13ffc1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a611e9f48f24ec1bdb3f8d0ec13ffc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a611e9f48f24ec1bdb3f8d0ec13ffc1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34969 } } }
I20260620 23:40:55.325685 21207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a611e9f48f24ec1bdb3f8d0ec13ffc1 [sys.catalog]: This master's current role is: LEADER
I20260620 23:40:55.333426 21216 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:40:55.342053 21216 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:40:55.346634 21204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:55.322257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36835 (local address 127.0.0.1:34969)
0620 23:40:55.322435 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:55.322441 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:55.323031 (+ 590us) server_negotiation.cc:407] Connection header received
0620 23:40:55.323357 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:55.323363 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:55.323448 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:55.323548 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:40:55.325860 (+ 2312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:55.327210 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:55.329852 (+ 2642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:55.330800 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:55.330964 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:55.338419 (+ 7455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:40:55.338427 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:40:55.338432 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:40:55.338494 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:40:55.342755 (+ 4261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:40:55.343899 (+ 1144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:40:55.343906 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:40:55.343909 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:40:55.344068 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:40:55.344577 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:40:55.344583 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:40:55.344587 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:40:55.345052 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:40:55.345299 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:55.345936 (+ 637us) server_negotiation.cc:299] Negotiation successful
0620 23:40:55.346217 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 23:40:55.354769 21216 catalog_manager.cc:1347] Generated new cluster ID: f469d69dedc64d0a9c3df662529143a2
I20260620 23:40:55.354905 21216 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:40:55.368889 21216 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:40:55.370318 21216 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:40:55.379426 21216 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a611e9f48f24ec1bdb3f8d0ec13ffc1: Generated new TSK 0
I20260620 23:40:55.380370 21216 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:40:55.388329 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 21134
I20260620 23:40:55.389698 21196 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:40:55.483646 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 21134
2026-06-20T23:40:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2035 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781998525501163-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:40:55 dist-test-slave-tcgx krb5kdc[21234](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:40:55 dist-test-slave-tcgx krb5kdc[21234](info): set up 2 sockets
Jun 20 23:40:55 dist-test-slave-tcgx krb5kdc[21234](info): commencing operation
krb5kdc: starting...
W20260620 23:40:56.607393 388 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.001s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781998525501163-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781998525501163-388-0/krb5kdc/test-user.keytab.
Jun 20 23:40:56 dist-test-slave-tcgx krb5kdc[21234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:40:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:40:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781998525501163-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781998525501163-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:40:56.874311 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781998525501163-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781998525501163-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781998525501163-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33619
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781998525501163-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33619
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:40:57.210520 21247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:40:57.210830 21247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:40:57.211021 21247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:40:57.218271 21247 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:40:57.218389 21247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:40:57.218502 21247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:40:57.218622 21247 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:40:57.218684 21247 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:40:57.228951 21247 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781998525501163-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781998525501163-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33619
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781998525501163-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781998525501163-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:40:57.230321 21247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:40:57.231722 21247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:40:57.243810 21252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:40:57.243860 21253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:40:57.243881 21255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:40:57.244730 21247 server_base.cc:1034] running on GCE node
I20260620 23:40:57.245587 21247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:40:57.247148 21247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:40:57.248380 21247 hybrid_clock.cc:648] HybridClock initialized: now 1781998857248332 us; error 44 us; skew 500 ppm
I20260620 23:40:57.248732 21247 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:40:57 dist-test-slave-tcgx krb5kdc[21234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998857, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:40:57.255537 21247 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:40:57.257107 21247 webserver.cc:460] Webserver started at http://127.0.0.1:33045/ using document root <none> and password file <none>
I20260620 23:40:57.257795 21247 fs_manager.cc:359] Metadata directory not provided
I20260620 23:40:57.257903 21247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:40:57.258178 21247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:40:57.261335 21247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781998525501163-388-0/minicluster-data/master-0/data/instance:
uuid: "b288bebf7b064c36a3b8554a3835242a"
format_stamp: "Formatted at 2026-06-20 23:40:57 on dist-test-slave-tcgx"
server_key: "3d82a4934ba7c1c29deecf908a66dbea"
server_key_iv: "d829f8fbd914581ffc08a5cbf49b8cb7"
server_key_version: "encryptionkey@0"
I20260620 23:40:57.262084 21247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781998525501163-388-0/minicluster-data/master-0/wal/instance:
uuid: "b288bebf7b064c36a3b8554a3835242a"
format_stamp: "Formatted at 2026-06-20 23:40:57 on dist-test-slave-tcgx"
server_key: "3d82a4934ba7c1c29deecf908a66dbea"
server_key_iv: "d829f8fbd914581ffc08a5cbf49b8cb7"
server_key_version: "encryptionkey@0"
I20260620 23:40:57.267700 21247 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260620 23:40:57.271332 21262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:57.272828 21247 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:40:57.273030 21247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781998525501163-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781998525501163-388-0/minicluster-data/master-0/wal
uuid: "b288bebf7b064c36a3b8554a3835242a"
format_stamp: "Formatted at 2026-06-20 23:40:57 on dist-test-slave-tcgx"
server_key: "3d82a4934ba7c1c29deecf908a66dbea"
server_key_iv: "d829f8fbd914581ffc08a5cbf49b8cb7"
server_key_version: "encryptionkey@0"
I20260620 23:40:57.273231 21247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781998525501163-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781998525501163-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781998525501163-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:40:57.287242 21247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:40:57.291775 21247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:40:57.292078 21247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:40:57.311295 21247 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33619
I20260620 23:40:57.311318 21313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33619 every 8 connection(s)
I20260620 23:40:57.312744 21247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:40:57.316576 21314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:40:57.320858 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 21247
I20260620 23:40:57.321056 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781998525501163-388-0/minicluster-data/master-0/wal/instance
I20260620 23:40:57.321485 388 external_mini_cluster.cc:1428] Setting key 17a88eb9618debe8b7c4e5baa04cf1c0
I20260620 23:40:57.329357 21314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b288bebf7b064c36a3b8554a3835242a: Bootstrap starting.
I20260620 23:40:57.333257 21314 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b288bebf7b064c36a3b8554a3835242a: Neither blocks nor log segments found. Creating new log.
I20260620 23:40:57.334544 21314 log.cc:824] T 00000000000000000000000000000000 P b288bebf7b064c36a3b8554a3835242a: Log is configured to *not* fsync() on all Append() calls
I20260620 23:40:57.337919 21314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b288bebf7b064c36a3b8554a3835242a: No bootstrap required, opened a new log
I20260620 23:40:57.337956 21317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:57.323431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50001 (local address 127.0.0.1:33619)
0620 23:40:57.324267 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:57.324284 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:57.324338 (+ 54us) server_negotiation.cc:407] Connection header received
0620 23:40:57.327456 (+ 3118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:57.327489 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:57.328073 (+ 584us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:57.328541 (+ 468us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:40:57.329081 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:57.331168 (+ 2087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:57.332610 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:57.333772 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:57.333954 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:57.335222 (+ 1268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:40:57.335231 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:40:57.335249 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:40:57.335872 (+ 623us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:40:57.335962 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:40:57.336007 (+ 45us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:40:57.336120 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:57.336332 (+ 212us) server_negotiation.cc:299] Negotiation successful
0620 23:40:57.336604 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":352,"threads_started":1}
I20260620 23:40:57.341759 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:40:57.346393 21314 raft_consensus.cc:348] T 00000000000000000000000000000000 P b288bebf7b064c36a3b8554a3835242a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b288bebf7b064c36a3b8554a3835242a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33619 } }
I20260620 23:40:57.346961 21314 raft_consensus.cc:374] T 00000000000000000000000000000000 P b288bebf7b064c36a3b8554a3835242a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:40:57.347084 21314 raft_consensus.cc:729] T 00000000000000000000000000000000 P b288bebf7b064c36a3b8554a3835242a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b288bebf7b064c36a3b8554a3835242a, State: Initialized, Role: FOLLOWER
I20260620 23:40:57.347842 21314 consensus_queue.cc:260] T 00000000000000000000000000000000 P b288bebf7b064c36a3b8554a3835242a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b288bebf7b064c36a3b8554a3835242a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33619 } }
I20260620 23:40:57.348150 21314 raft_consensus.cc:388] T 00000000000000000000000000000000 P b288bebf7b064c36a3b8554a3835242a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:40:57.348342 21314 raft_consensus.cc:482] T 00000000000000000000000000000000 P b288bebf7b064c36a3b8554a3835242a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:40:57.348549 21314 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b288bebf7b064c36a3b8554a3835242a [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:40:57.351755 21314 raft_consensus.cc:504] T 00000000000000000000000000000000 P b288bebf7b064c36a3b8554a3835242a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b288bebf7b064c36a3b8554a3835242a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33619 } }
I20260620 23:40:57.352398 21314 leader_election.cc:302] T 00000000000000000000000000000000 P b288bebf7b064c36a3b8554a3835242a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b288bebf7b064c36a3b8554a3835242a; no voters:
I20260620 23:40:57.353164 21314 leader_election.cc:288] T 00000000000000000000000000000000 P b288bebf7b064c36a3b8554a3835242a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:40:57.353271 21324 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b288bebf7b064c36a3b8554a3835242a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:40:57.353624 21317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:57.346266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55469 (local address 127.0.0.1:33619)
0620 23:40:57.346510 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:57.346517 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:57.346897 (+ 380us) server_negotiation.cc:407] Connection header received
0620 23:40:57.347158 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:57.347162 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:57.347259 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:57.347353 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:40:57.347977 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:57.348954 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:57.350826 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:40:57.351331 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:40:57.351426 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:40:57.352676 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:40:57.352681 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:40:57.352685 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:40:57.352702 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:40:57.352740 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:40:57.352755 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:40:57.352822 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:57.353044 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 23:40:57.353190 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 23:40:57.355645 21314 sys_catalog.cc:564] T 00000000000000000000000000000000 P b288bebf7b064c36a3b8554a3835242a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:40:57.359854 21324 raft_consensus.cc:686] T 00000000000000000000000000000000 P b288bebf7b064c36a3b8554a3835242a [term 1 LEADER]: Becoming Leader. State: Replica: b288bebf7b064c36a3b8554a3835242a, State: Running, Role: LEADER
I20260620 23:40:57.360749 21324 consensus_queue.cc:237] T 00000000000000000000000000000000 P b288bebf7b064c36a3b8554a3835242a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b288bebf7b064c36a3b8554a3835242a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33619 } }
I20260620 23:40:57.363786 21325 sys_catalog.cc:455] T 00000000000000000000000000000000 P b288bebf7b064c36a3b8554a3835242a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b288bebf7b064c36a3b8554a3835242a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b288bebf7b064c36a3b8554a3835242a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33619 } } }
I20260620 23:40:57.363967 21326 sys_catalog.cc:455] T 00000000000000000000000000000000 P b288bebf7b064c36a3b8554a3835242a [sys.catalog]: SysCatalogTable state changed. Reason: New leader b288bebf7b064c36a3b8554a3835242a. Latest consensus state: current_term: 1 leader_uuid: "b288bebf7b064c36a3b8554a3835242a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b288bebf7b064c36a3b8554a3835242a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33619 } } }
I20260620 23:40:57.364223 21326 sys_catalog.cc:458] T 00000000000000000000000000000000 P b288bebf7b064c36a3b8554a3835242a [sys.catalog]: This master's current role is: LEADER
I20260620 23:40:57.364223 21325 sys_catalog.cc:458] T 00000000000000000000000000000000 P b288bebf7b064c36a3b8554a3835242a [sys.catalog]: This master's current role is: LEADER
W20260620 23:40:57.370321 21338 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b288bebf7b064c36a3b8554a3835242a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:40:57.370458 21338 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:40:57.374902 21339 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:40:57.376932 21339 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:40:57.385957 21339 catalog_manager.cc:1347] Generated new cluster ID: 006aa98d17864f2a9d24c72a543d0328
I20260620 23:40:57.386104 21339 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:40:57.404639 21339 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:40:57.406615 21339 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:40:57.416888 21339 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b288bebf7b064c36a3b8554a3835242a: Generated new TSK 0
I20260620 23:40:57.418018 21339 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:40:57.426357 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 21247
W20260620 23:40:57.428153 21309 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:50001
W20260620 23:40:57.428829 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 21247: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:40:57.428984 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 21247
2026-06-20T23:40:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1936 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781998525501163-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:40:57 dist-test-slave-tcgx krb5kdc[21345](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:40:57 dist-test-slave-tcgx krb5kdc[21345](info): set up 2 sockets
Jun 20 23:40:57 dist-test-slave-tcgx krb5kdc[21345](info): commencing operation
krb5kdc: starting...
W20260620 23:40:58.547389 388 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781998525501163-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781998525501163-388-0/krb5kdc/test-user.keytab.
Jun 20 23:40:58 dist-test-slave-tcgx krb5kdc[21345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:40:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:40:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781998525501163-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781998525501163-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:40:58.811754 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781998525501163-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781998525501163-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781998525501163-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45223
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781998525501163-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45223
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:40:59.148365 21358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:40:59.148684 21358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:40:59.148779 21358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:40:59.156258 21358 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:40:59.156387 21358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:40:59.156436 21358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:40:59.156499 21358 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:40:59.156546 21358 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:40:59.167407 21358 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781998525501163-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781998525501163-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45223
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781998525501163-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781998525501163-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:40:59.168784 21358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:40:59.170102 21358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:40:59.182498 21363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:40:59.182600 21364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:40:59.182546 21366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:40:59.183573 21358 server_base.cc:1034] running on GCE node
I20260620 23:40:59.184504 21358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:40:59.186080 21358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:40:59.187331 21358 hybrid_clock.cc:648] HybridClock initialized: now 1781998859187261 us; error 58 us; skew 500 ppm
I20260620 23:40:59.187690 21358 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:40:59 dist-test-slave-tcgx krb5kdc[21345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998859, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:40:59.190913 21358 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:40:59.192564 21358 webserver.cc:460] Webserver started at http://127.0.0.1:44437/ using document root <none> and password file <none>
I20260620 23:40:59.193300 21358 fs_manager.cc:359] Metadata directory not provided
I20260620 23:40:59.193450 21358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:40:59.193768 21358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:40:59.197002 21358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781998525501163-388-0/minicluster-data/master-0/data/instance:
uuid: "91a5da4e73f846deb89eaedf26212cfe"
format_stamp: "Formatted at 2026-06-20 23:40:59 on dist-test-slave-tcgx"
server_key: "53daad8839ff072513bc7564d61178d8"
server_key_iv: "f7f3d9de308144ab37568a8106119eea"
server_key_version: "encryptionkey@0"
I20260620 23:40:59.197815 21358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781998525501163-388-0/minicluster-data/master-0/wal/instance:
uuid: "91a5da4e73f846deb89eaedf26212cfe"
format_stamp: "Formatted at 2026-06-20 23:40:59 on dist-test-slave-tcgx"
server_key: "53daad8839ff072513bc7564d61178d8"
server_key_iv: "f7f3d9de308144ab37568a8106119eea"
server_key_version: "encryptionkey@0"
I20260620 23:40:59.203380 21358 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260620 23:40:59.207027 21373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:40:59.208487 21358 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:40:59.208717 21358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781998525501163-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781998525501163-388-0/minicluster-data/master-0/wal
uuid: "91a5da4e73f846deb89eaedf26212cfe"
format_stamp: "Formatted at 2026-06-20 23:40:59 on dist-test-slave-tcgx"
server_key: "53daad8839ff072513bc7564d61178d8"
server_key_iv: "f7f3d9de308144ab37568a8106119eea"
server_key_version: "encryptionkey@0"
I20260620 23:40:59.208938 21358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781998525501163-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781998525501163-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781998525501163-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:40:59.214730 21358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:40:59.219173 21358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:40:59.219475 21358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:40:59.239058 21358 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45223
I20260620 23:40:59.239100 21424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45223 every 8 connection(s)
I20260620 23:40:59.240526 21358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:40:59.244505 21425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:40:59.246872 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 21358
I20260620 23:40:59.247161 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781998525501163-388-0/minicluster-data/master-0/wal/instance
I20260620 23:40:59.247625 388 external_mini_cluster.cc:1428] Setting key 79f087a213d52d0f39965f4efc3b52f2
I20260620 23:40:59.257361 21428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:59.249717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38029 (local address 127.0.0.1:45223)
0620 23:40:59.250599 (+ 882us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:59.250617 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:59.250682 (+ 65us) server_negotiation.cc:407] Connection header received
0620 23:40:59.253094 (+ 2412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:59.253128 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:59.253626 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:59.254154 (+ 528us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:40:59.255223 (+ 1069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:40:59.255238 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:40:59.255257 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:40:59.255275 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:40:59.255356 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:40:59.255394 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:40:59.255629 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:59.255788 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 23:40:59.256010 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":368,"threads_started":1}
I20260620 23:40:59.258589 21425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91a5da4e73f846deb89eaedf26212cfe: Bootstrap starting.
I20260620 23:40:59.261391 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:40:59.262197 21425 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91a5da4e73f846deb89eaedf26212cfe: Neither blocks nor log segments found. Creating new log.
I20260620 23:40:59.263361 21425 log.cc:824] T 00000000000000000000000000000000 P 91a5da4e73f846deb89eaedf26212cfe: Log is configured to *not* fsync() on all Append() calls
I20260620 23:40:59.267035 21425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91a5da4e73f846deb89eaedf26212cfe: No bootstrap required, opened a new log
I20260620 23:40:59.268589 21428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:40:59.265917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51499 (local address 127.0.0.1:45223)
0620 23:40:59.266108 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 23:40:59.266114 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:40:59.266499 (+ 385us) server_negotiation.cc:407] Connection header received
0620 23:40:59.266710 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:40:59.266722 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:40:59.266774 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:40:59.266882 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:40:59.267918 (+ 1036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:40:59.267924 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:40:59.267928 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:40:59.267947 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:40:59.268002 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:40:59.268018 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:40:59.268150 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 23:40:59.268247 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 23:40:59.268311 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 23:40:59.275399 21425 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91a5da4e73f846deb89eaedf26212cfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91a5da4e73f846deb89eaedf26212cfe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45223 } }
I20260620 23:40:59.275913 21425 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91a5da4e73f846deb89eaedf26212cfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:40:59.276024 21425 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91a5da4e73f846deb89eaedf26212cfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91a5da4e73f846deb89eaedf26212cfe, State: Initialized, Role: FOLLOWER
I20260620 23:40:59.276626 21425 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91a5da4e73f846deb89eaedf26212cfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91a5da4e73f846deb89eaedf26212cfe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45223 } }
I20260620 23:40:59.276876 21425 raft_consensus.cc:388] T 00000000000000000000000000000000 P 91a5da4e73f846deb89eaedf26212cfe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:40:59.277040 21425 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91a5da4e73f846deb89eaedf26212cfe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:40:59.277227 21425 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91a5da4e73f846deb89eaedf26212cfe [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:40:59.279551 21425 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91a5da4e73f846deb89eaedf26212cfe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91a5da4e73f846deb89eaedf26212cfe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45223 } }
I20260620 23:40:59.280139 21425 leader_election.cc:302] T 00000000000000000000000000000000 P 91a5da4e73f846deb89eaedf26212cfe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 91a5da4e73f846deb89eaedf26212cfe; no voters:
I20260620 23:40:59.280777 21425 leader_election.cc:288] T 00000000000000000000000000000000 P 91a5da4e73f846deb89eaedf26212cfe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:40:59.280893 21435 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91a5da4e73f846deb89eaedf26212cfe [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:40:59.282248 21435 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91a5da4e73f846deb89eaedf26212cfe [term 1 LEADER]: Becoming Leader. State: Replica: 91a5da4e73f846deb89eaedf26212cfe, State: Running, Role: LEADER
I20260620 23:40:59.282768 21435 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91a5da4e73f846deb89eaedf26212cfe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91a5da4e73f846deb89eaedf26212cfe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45223 } }
I20260620 23:40:59.283151 21425 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91a5da4e73f846deb89eaedf26212cfe [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:40:59.285881 21436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91a5da4e73f846deb89eaedf26212cfe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "91a5da4e73f846deb89eaedf26212cfe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91a5da4e73f846deb89eaedf26212cfe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45223 } } }
I20260620 23:40:59.285856 21437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91a5da4e73f846deb89eaedf26212cfe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91a5da4e73f846deb89eaedf26212cfe. Latest consensus state: current_term: 1 leader_uuid: "91a5da4e73f846deb89eaedf26212cfe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91a5da4e73f846deb89eaedf26212cfe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45223 } } }
I20260620 23:40:59.286396 21436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91a5da4e73f846deb89eaedf26212cfe [sys.catalog]: This master's current role is: LEADER
I20260620 23:40:59.286397 21437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91a5da4e73f846deb89eaedf26212cfe [sys.catalog]: This master's current role is: LEADER
I20260620 23:40:59.287375 21442 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:40:59.293685 21442 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:40:59.303076 21442 catalog_manager.cc:1347] Generated new cluster ID: 950c3dc23f78464dbe29d0081a5a683a
I20260620 23:40:59.303215 21442 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:40:59.335623 21442 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:40:59.336098 21442 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:40:59.345031 21442 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 91a5da4e73f846deb89eaedf26212cfe: Generated new TSK 0
I20260620 23:40:59.346208 21442 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:40:59.361280 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 21358
W20260620 23:40:59.362603 21420 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38029
W20260620 23:40:59.363451 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 21358: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:40:59.363615 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 21358
2026-06-20T23:40:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1934 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781998525501163-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:40:59 dist-test-slave-tcgx krb5kdc[21456](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:40:59 dist-test-slave-tcgx krb5kdc[21456](info): set up 2 sockets
Jun 20 23:40:59 dist-test-slave-tcgx krb5kdc[21456](info): commencing operation
krb5kdc: starting...
W20260620 23:41:00.471546 388 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.004s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781998525501163-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781998525501163-388-0/krb5kdc/test-user.keytab.
Jun 20 23:41:00 dist-test-slave-tcgx krb5kdc[21456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:41:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:41:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781998525501163-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781998525501163-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:41:00.733861 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781998525501163-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781998525501163-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781998525501163-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42759
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781998525501163-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42759
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:41:01.075913 21469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:41:01.076248 21469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:41:01.076419 21469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:41:01.083845 21469 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:41:01.083967 21469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:41:01.084103 21469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:41:01.084216 21469 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:41:01.084300 21469 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:41:01.094441 21469 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781998525501163-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781998525501163-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42759
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781998525501163-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781998525501163-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:41:01.095950 21469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:41:01.097334 21469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:41:01.110004 21474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:41:01.109927 21477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:41:01.110023 21475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:41:01.111093 21469 server_base.cc:1034] running on GCE node
I20260620 23:41:01.112167 21469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:41:01.113951 21469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:41:01.115321 21469 hybrid_clock.cc:648] HybridClock initialized: now 1781998861115287 us; error 78 us; skew 500 ppm
I20260620 23:41:01.115751 21469 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:41:01 dist-test-slave-tcgx krb5kdc[21456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998861, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:41:01.119316 21469 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:41:01.121012 21469 webserver.cc:460] Webserver started at http://127.0.0.1:33173/ using document root <none> and password file <none>
I20260620 23:41:01.121768 21469 fs_manager.cc:359] Metadata directory not provided
I20260620 23:41:01.121920 21469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:41:01.122246 21469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:41:01.125526 21469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781998525501163-388-0/minicluster-data/master-0/data/instance:
uuid: "ed13fdaf83c5472a9f7093c23c7bcb78"
format_stamp: "Formatted at 2026-06-20 23:41:01 on dist-test-slave-tcgx"
server_key: "303e4f8a898b3479f145c86d808f7ebd"
server_key_iv: "89bd1303cadebe81e0c0531d9ccc930f"
server_key_version: "encryptionkey@0"
I20260620 23:41:01.126344 21469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781998525501163-388-0/minicluster-data/master-0/wal/instance:
uuid: "ed13fdaf83c5472a9f7093c23c7bcb78"
format_stamp: "Formatted at 2026-06-20 23:41:01 on dist-test-slave-tcgx"
server_key: "303e4f8a898b3479f145c86d808f7ebd"
server_key_iv: "89bd1303cadebe81e0c0531d9ccc930f"
server_key_version: "encryptionkey@0"
I20260620 23:41:01.131868 21469 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 23:41:01.135758 21484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:41:01.137259 21469 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:41:01.137454 21469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781998525501163-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781998525501163-388-0/minicluster-data/master-0/wal
uuid: "ed13fdaf83c5472a9f7093c23c7bcb78"
format_stamp: "Formatted at 2026-06-20 23:41:01 on dist-test-slave-tcgx"
server_key: "303e4f8a898b3479f145c86d808f7ebd"
server_key_iv: "89bd1303cadebe81e0c0531d9ccc930f"
server_key_version: "encryptionkey@0"
I20260620 23:41:01.137674 21469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781998525501163-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781998525501163-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781998525501163-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:41:01.155217 21469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:41:01.159520 21469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:41:01.159830 21469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:41:01.179317 21469 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42759
I20260620 23:41:01.179361 21535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42759 every 8 connection(s)
I20260620 23:41:01.180764 21469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:41:01.184985 21536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:41:01.190707 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 21469
I20260620 23:41:01.190972 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781998525501163-388-0/minicluster-data/master-0/wal/instance
I20260620 23:41:01.191577 388 external_mini_cluster.cc:1428] Setting key 1a1465a0a3a11e53db6fe247aaa55497
I20260620 23:41:01.198099 21536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed13fdaf83c5472a9f7093c23c7bcb78: Bootstrap starting.
I20260620 23:41:01.202239 21536 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed13fdaf83c5472a9f7093c23c7bcb78: Neither blocks nor log segments found. Creating new log.
I20260620 23:41:01.203645 21536 log.cc:824] T 00000000000000000000000000000000 P ed13fdaf83c5472a9f7093c23c7bcb78: Log is configured to *not* fsync() on all Append() calls
I20260620 23:41:01.206852 21536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed13fdaf83c5472a9f7093c23c7bcb78: No bootstrap required, opened a new log
Jun 20 23:41:01 dist-test-slave-tcgx krb5kdc[21456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 23:41:01.219076 21536 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed13fdaf83c5472a9f7093c23c7bcb78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed13fdaf83c5472a9f7093c23c7bcb78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42759 } }
I20260620 23:41:01.219512 21539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:01.193671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.182:44787 (local address 127.0.0.1:42759)
0620 23:41:01.194686 (+ 1015us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:01.194700 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:01.194744 (+ 44us) server_negotiation.cc:407] Connection header received
0620 23:41:01.196787 (+ 2043us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:01.196814 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:01.197320 (+ 506us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:01.197704 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:01.199720 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:01.201240 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:01.203292 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:01.204149 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:01.204385 (+ 236us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:01.209266 (+ 4881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:41:01.209280 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:41:01.209302 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:41:01.209359 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:41:01.214084 (+ 4725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:01.215049 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:01.215059 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:01.215069 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:01.215317 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:01.215790 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:01.215795 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:01.215799 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:01.216655 (+ 856us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:41:01.217056 (+ 401us) server_negotiation.cc:1035] Waiting for connection context
0620 23:41:01.217502 (+ 446us) server_negotiation.cc:299] Negotiation successful
0620 23:41:01.217928 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":762,"thread_start_us":506,"threads_started":1}
I20260620 23:41:01.219687 21536 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed13fdaf83c5472a9f7093c23c7bcb78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:41:01.219794 21536 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed13fdaf83c5472a9f7093c23c7bcb78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed13fdaf83c5472a9f7093c23c7bcb78, State: Initialized, Role: FOLLOWER
I20260620 23:41:01.220587 21536 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed13fdaf83c5472a9f7093c23c7bcb78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed13fdaf83c5472a9f7093c23c7bcb78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42759 } }
I20260620 23:41:01.220916 21536 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed13fdaf83c5472a9f7093c23c7bcb78 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:41:01.221078 21536 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed13fdaf83c5472a9f7093c23c7bcb78 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:41:01.221288 21536 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed13fdaf83c5472a9f7093c23c7bcb78 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:41:01.224056 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:41:01.224493 21536 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed13fdaf83c5472a9f7093c23c7bcb78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed13fdaf83c5472a9f7093c23c7bcb78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42759 } }
I20260620 23:41:01.225064 21536 leader_election.cc:302] T 00000000000000000000000000000000 P ed13fdaf83c5472a9f7093c23c7bcb78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed13fdaf83c5472a9f7093c23c7bcb78; no voters:
I20260620 23:41:01.225723 21536 leader_election.cc:288] T 00000000000000000000000000000000 P ed13fdaf83c5472a9f7093c23c7bcb78 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:41:01.225881 21541 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed13fdaf83c5472a9f7093c23c7bcb78 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:41:01.226575 21541 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed13fdaf83c5472a9f7093c23c7bcb78 [term 1 LEADER]: Becoming Leader. State: Replica: ed13fdaf83c5472a9f7093c23c7bcb78, State: Running, Role: LEADER
I20260620 23:41:01.227391 21541 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed13fdaf83c5472a9f7093c23c7bcb78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed13fdaf83c5472a9f7093c23c7bcb78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42759 } }
I20260620 23:41:01.230019 21541 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed13fdaf83c5472a9f7093c23c7bcb78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed13fdaf83c5472a9f7093c23c7bcb78. Latest consensus state: current_term: 1 leader_uuid: "ed13fdaf83c5472a9f7093c23c7bcb78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed13fdaf83c5472a9f7093c23c7bcb78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42759 } } }
I20260620 23:41:01.231946 21541 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed13fdaf83c5472a9f7093c23c7bcb78 [sys.catalog]: This master's current role is: LEADER
I20260620 23:41:01.231501 21543 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed13fdaf83c5472a9f7093c23c7bcb78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed13fdaf83c5472a9f7093c23c7bcb78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed13fdaf83c5472a9f7093c23c7bcb78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42759 } } }
I20260620 23:41:01.232355 21543 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed13fdaf83c5472a9f7093c23c7bcb78 [sys.catalog]: This master's current role is: LEADER
I20260620 23:41:01.241336 21536 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed13fdaf83c5472a9f7093c23c7bcb78 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:41:01.241605 21546 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:41:01.249070 21546 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:41:01.257921 21563 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 23:41:01.259011 21539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:01.239723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.182:41001 (local address 127.0.0.1:42759)
0620 23:41:01.239944 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:01.239952 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:01.241170 (+ 1218us) server_negotiation.cc:407] Connection header received
0620 23:41:01.241402 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:01.241408 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:01.241492 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:01.241597 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:01.243340 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:01.244353 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:01.246233 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:01.246959 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:01.247098 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:01.250719 (+ 3621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:41:01.250726 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:41:01.250730 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:41:01.250773 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:41:01.254842 (+ 4069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:01.256176 (+ 1334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:01.256181 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:01.256184 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:01.256338 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:01.256938 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:01.256943 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:01.256946 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:01.257422 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:41:01.257703 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0620 23:41:01.258392 (+ 689us) server_negotiation.cc:299] Negotiation successful
0620 23:41:01.258596 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 23:41:01.265740 21546 catalog_manager.cc:1347] Generated new cluster ID: 68062c914f6b4cd7b172ea9a8aaecc90
I20260620 23:41:01.265931 21546 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:41:01.273284 21546 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:41:01.274577 21546 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:41:01.284541 21546 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed13fdaf83c5472a9f7093c23c7bcb78: Generated new TSK 0
I20260620 23:41:01.285476 21546 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:41:01.295189 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 21469
I20260620 23:41:01.297294 21531 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:41:01.392163 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 21469
W20260620 23:41:01.411890 21454 connection.cc:535] client connection to 127.0.0.1:42759 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42759): Connection reset by peer (error 104)
2026-06-20T23:41:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2030 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781998525501163-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:41:01 dist-test-slave-tcgx krb5kdc[21568](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:41:01 dist-test-slave-tcgx krb5kdc[21568](info): set up 2 sockets
Jun 20 23:41:01 dist-test-slave-tcgx krb5kdc[21568](info): commencing operation
krb5kdc: starting...
W20260620 23:41:02.503599 388 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781998525501163-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781998525501163-388-0/krb5kdc/test-user.keytab.
Jun 20 23:41:02 dist-test-slave-tcgx krb5kdc[21568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:41:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:41:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781998525501163-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781998525501163-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:41:02.779171 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781998525501163-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781998525501163-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781998525501163-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43643
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781998525501163-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43643
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:41:03.124043 21581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:41:03.124444 21581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:41:03.124626 21581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:41:03.132406 21581 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:41:03.132596 21581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:41:03.132716 21581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:41:03.132825 21581 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:41:03.132911 21581 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:41:03.143440 21581 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781998525501163-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781998525501163-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43643
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781998525501163-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781998525501163-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:41:03.144903 21581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:41:03.146327 21581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:41:03.158989 21586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:41:03.159114 21587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:41:03.159057 21589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:41:03.160583 21581 server_base.cc:1034] running on GCE node
I20260620 23:41:03.161463 21581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:41:03.163123 21581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:41:03.164374 21581 hybrid_clock.cc:648] HybridClock initialized: now 1781998863164329 us; error 42 us; skew 500 ppm
I20260620 23:41:03.164773 21581 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:41:03 dist-test-slave-tcgx krb5kdc[21568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998863, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:41:03.172884 21581 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:41:03.174636 21581 webserver.cc:460] Webserver started at http://127.0.0.1:44799/ using document root <none> and password file <none>
I20260620 23:41:03.175479 21581 fs_manager.cc:359] Metadata directory not provided
I20260620 23:41:03.175637 21581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:41:03.175964 21581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:41:03.179186 21581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781998525501163-388-0/minicluster-data/master-0/data/instance:
uuid: "f038e8d4a1a24de5b29437a9adf122a9"
format_stamp: "Formatted at 2026-06-20 23:41:03 on dist-test-slave-tcgx"
server_key: "f58164b3876d5083624a543f2f5ea41a"
server_key_iv: "9c9cf79b0b1b76eb857f7c0761384ed8"
server_key_version: "encryptionkey@0"
I20260620 23:41:03.180023 21581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781998525501163-388-0/minicluster-data/master-0/wal/instance:
uuid: "f038e8d4a1a24de5b29437a9adf122a9"
format_stamp: "Formatted at 2026-06-20 23:41:03 on dist-test-slave-tcgx"
server_key: "f58164b3876d5083624a543f2f5ea41a"
server_key_iv: "9c9cf79b0b1b76eb857f7c0761384ed8"
server_key_version: "encryptionkey@0"
I20260620 23:41:03.185600 21581 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:41:03.189334 21596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:41:03.190774 21581 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:41:03.191047 21581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781998525501163-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781998525501163-388-0/minicluster-data/master-0/wal
uuid: "f038e8d4a1a24de5b29437a9adf122a9"
format_stamp: "Formatted at 2026-06-20 23:41:03 on dist-test-slave-tcgx"
server_key: "f58164b3876d5083624a543f2f5ea41a"
server_key_iv: "9c9cf79b0b1b76eb857f7c0761384ed8"
server_key_version: "encryptionkey@0"
I20260620 23:41:03.191274 21581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781998525501163-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781998525501163-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781998525501163-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:41:03.211977 21581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:41:03.216643 21581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:41:03.216960 21581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:41:03.237056 21581 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43643
I20260620 23:41:03.237157 21647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43643 every 8 connection(s)
I20260620 23:41:03.238559 21581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:41:03.242815 21648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:41:03.247501 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 21581
I20260620 23:41:03.247714 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781998525501163-388-0/minicluster-data/master-0/wal/instance
I20260620 23:41:03.248189 388 external_mini_cluster.cc:1428] Setting key dfab4e99ad477aa948607e1505748e30
I20260620 23:41:03.257594 21648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f038e8d4a1a24de5b29437a9adf122a9: Bootstrap starting.
I20260620 23:41:03.261868 21648 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f038e8d4a1a24de5b29437a9adf122a9: Neither blocks nor log segments found. Creating new log.
I20260620 23:41:03.262799 21651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:03.250320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.182:44113 (local address 127.0.0.1:43643)
0620 23:41:03.251136 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:03.251149 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:03.251194 (+ 45us) server_negotiation.cc:407] Connection header received
0620 23:41:03.252877 (+ 1683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:03.252902 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:03.253392 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:03.253758 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:03.254401 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:03.255848 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:03.257953 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:03.258660 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:03.258796 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:03.260350 (+ 1554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:41:03.260358 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:41:03.260373 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:41:03.260921 (+ 548us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:41:03.260999 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:41:03.261023 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:41:03.261142 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 23:41:03.261448 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 23:41:03.261665 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":347,"threads_started":1}
I20260620 23:41:03.263186 21648 log.cc:824] T 00000000000000000000000000000000 P f038e8d4a1a24de5b29437a9adf122a9: Log is configured to *not* fsync() on all Append() calls
I20260620 23:41:03.266708 21648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f038e8d4a1a24de5b29437a9adf122a9: No bootstrap required, opened a new log
I20260620 23:41:03.267128 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:41:03.276239 21648 raft_consensus.cc:348] T 00000000000000000000000000000000 P f038e8d4a1a24de5b29437a9adf122a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f038e8d4a1a24de5b29437a9adf122a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43643 } }
I20260620 23:41:03.276718 21648 raft_consensus.cc:374] T 00000000000000000000000000000000 P f038e8d4a1a24de5b29437a9adf122a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:41:03.276861 21648 raft_consensus.cc:729] T 00000000000000000000000000000000 P f038e8d4a1a24de5b29437a9adf122a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f038e8d4a1a24de5b29437a9adf122a9, State: Initialized, Role: FOLLOWER
I20260620 23:41:03.277614 21648 consensus_queue.cc:260] T 00000000000000000000000000000000 P f038e8d4a1a24de5b29437a9adf122a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f038e8d4a1a24de5b29437a9adf122a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43643 } }
I20260620 23:41:03.277895 21648 raft_consensus.cc:388] T 00000000000000000000000000000000 P f038e8d4a1a24de5b29437a9adf122a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:41:03.278097 21648 raft_consensus.cc:482] T 00000000000000000000000000000000 P f038e8d4a1a24de5b29437a9adf122a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:41:03.278232 21651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:03.271774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.182:40391 (local address 127.0.0.1:43643)
0620 23:41:03.271982 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:03.271991 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:03.272336 (+ 345us) server_negotiation.cc:407] Connection header received
0620 23:41:03.272569 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:03.272573 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:03.272643 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:03.272766 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:03.273309 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:03.274105 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:03.275562 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:03.276015 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:03.276149 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:03.277384 (+ 1235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:41:03.277388 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:41:03.277391 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:41:03.277410 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:41:03.277450 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:41:03.277464 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:41:03.277593 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 23:41:03.277801 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 23:41:03.277859 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 23:41:03.278344 21648 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f038e8d4a1a24de5b29437a9adf122a9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:41:03.281297 21648 raft_consensus.cc:504] T 00000000000000000000000000000000 P f038e8d4a1a24de5b29437a9adf122a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f038e8d4a1a24de5b29437a9adf122a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43643 } }
I20260620 23:41:03.281958 21648 leader_election.cc:302] T 00000000000000000000000000000000 P f038e8d4a1a24de5b29437a9adf122a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f038e8d4a1a24de5b29437a9adf122a9; no voters:
I20260620 23:41:03.282652 21648 leader_election.cc:288] T 00000000000000000000000000000000 P f038e8d4a1a24de5b29437a9adf122a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:41:03.282800 21658 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f038e8d4a1a24de5b29437a9adf122a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:41:03.283633 21658 raft_consensus.cc:686] T 00000000000000000000000000000000 P f038e8d4a1a24de5b29437a9adf122a9 [term 1 LEADER]: Becoming Leader. State: Replica: f038e8d4a1a24de5b29437a9adf122a9, State: Running, Role: LEADER
I20260620 23:41:03.284274 21658 consensus_queue.cc:237] T 00000000000000000000000000000000 P f038e8d4a1a24de5b29437a9adf122a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f038e8d4a1a24de5b29437a9adf122a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43643 } }
I20260620 23:41:03.285084 21648 sys_catalog.cc:564] T 00000000000000000000000000000000 P f038e8d4a1a24de5b29437a9adf122a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:41:03.287336 21659 sys_catalog.cc:455] T 00000000000000000000000000000000 P f038e8d4a1a24de5b29437a9adf122a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f038e8d4a1a24de5b29437a9adf122a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f038e8d4a1a24de5b29437a9adf122a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43643 } } }
I20260620 23:41:03.287878 21659 sys_catalog.cc:458] T 00000000000000000000000000000000 P f038e8d4a1a24de5b29437a9adf122a9 [sys.catalog]: This master's current role is: LEADER
I20260620 23:41:03.289171 21660 sys_catalog.cc:455] T 00000000000000000000000000000000 P f038e8d4a1a24de5b29437a9adf122a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f038e8d4a1a24de5b29437a9adf122a9. Latest consensus state: current_term: 1 leader_uuid: "f038e8d4a1a24de5b29437a9adf122a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f038e8d4a1a24de5b29437a9adf122a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43643 } } }
I20260620 23:41:03.291662 21660 sys_catalog.cc:458] T 00000000000000000000000000000000 P f038e8d4a1a24de5b29437a9adf122a9 [sys.catalog]: This master's current role is: LEADER
I20260620 23:41:03.292778 21666 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:41:03.299669 21666 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:41:03.309746 21666 catalog_manager.cc:1347] Generated new cluster ID: 7deda45d8f074b7da93b1928ffd19606
I20260620 23:41:03.309885 21666 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:41:03.327476 21666 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:41:03.328878 21666 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:41:03.339355 21666 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f038e8d4a1a24de5b29437a9adf122a9: Generated new TSK 0
I20260620 23:41:03.340329 21666 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:41:03.349313 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 21581
W20260620 23:41:03.350577 21643 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.182:44113
W20260620 23:41:03.351450 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 21581: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:41:03.351614 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 21581
W20260620 23:41:03.368966 21566 connection.cc:535] client connection to 127.0.0.1:43643 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43643): Connection reset by peer (error 104)
2026-06-20T23:41:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1957 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781998525501163-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:41:03 dist-test-slave-tcgx krb5kdc[21679](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:41:03 dist-test-slave-tcgx krb5kdc[21679](info): set up 2 sockets
Jun 20 23:41:03 dist-test-slave-tcgx krb5kdc[21679](info): commencing operation
krb5kdc: starting...
W20260620 23:41:04.536181 388 mini_kdc.cc:121] Time spent starting KDC: real 1.146s user 0.001s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781998525501163-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781998525501163-388-0/krb5kdc/test-user.keytab.
Jun 20 23:41:04 dist-test-slave-tcgx krb5kdc[21679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:41:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:41:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781998525501163-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781998525501163-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:41:04.812760 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781998525501163-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781998525501163-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781998525501163-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41681
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781998525501163-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41681
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:41:05.158126 21692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:41:05.158427 21692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:41:05.158574 21692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:41:05.166095 21692 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:41:05.166227 21692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:41:05.166312 21692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:41:05.166411 21692 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:41:05.166487 21692 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:41:05.176909 21692 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781998525501163-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781998525501163-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41681
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781998525501163-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781998525501163-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:41:05.178387 21692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:41:05.179800 21692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:41:05.191865 21700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:41:05.191912 21698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:41:05.191821 21697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:41:05.193277 21692 server_base.cc:1034] running on GCE node
I20260620 23:41:05.194119 21692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:41:05.195772 21692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:41:05.197184 21692 hybrid_clock.cc:648] HybridClock initialized: now 1781998865197123 us; error 50 us; skew 500 ppm
I20260620 23:41:05.197608 21692 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:41:05 dist-test-slave-tcgx krb5kdc[21679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998865, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:41:05.201442 21692 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:41:05.203120 21692 webserver.cc:460] Webserver started at http://127.0.0.1:35601/ using document root <none> and password file <none>
I20260620 23:41:05.203876 21692 fs_manager.cc:359] Metadata directory not provided
I20260620 23:41:05.204025 21692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:41:05.204371 21692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:41:05.207641 21692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781998525501163-388-0/minicluster-data/master-0/data/instance:
uuid: "f24d5e83025b402f8691b8c9014ab169"
format_stamp: "Formatted at 2026-06-20 23:41:05 on dist-test-slave-tcgx"
server_key: "40b342acca4c8ae2d757501dbe494c4b"
server_key_iv: "d7f96d6b2fea8ea99f889c68c7b6e6f4"
server_key_version: "encryptionkey@0"
I20260620 23:41:05.208482 21692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781998525501163-388-0/minicluster-data/master-0/wal/instance:
uuid: "f24d5e83025b402f8691b8c9014ab169"
format_stamp: "Formatted at 2026-06-20 23:41:05 on dist-test-slave-tcgx"
server_key: "40b342acca4c8ae2d757501dbe494c4b"
server_key_iv: "d7f96d6b2fea8ea99f889c68c7b6e6f4"
server_key_version: "encryptionkey@0"
I20260620 23:41:05.214036 21692 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 23:41:05.217824 21707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:41:05.219384 21692 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:41:05.219586 21692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781998525501163-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781998525501163-388-0/minicluster-data/master-0/wal
uuid: "f24d5e83025b402f8691b8c9014ab169"
format_stamp: "Formatted at 2026-06-20 23:41:05 on dist-test-slave-tcgx"
server_key: "40b342acca4c8ae2d757501dbe494c4b"
server_key_iv: "d7f96d6b2fea8ea99f889c68c7b6e6f4"
server_key_version: "encryptionkey@0"
I20260620 23:41:05.219894 21692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781998525501163-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781998525501163-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781998525501163-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:41:05.225827 21692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:41:05.230093 21692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:41:05.230423 21692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:41:05.249388 21692 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41681
I20260620 23:41:05.249437 21758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41681 every 8 connection(s)
I20260620 23:41:05.250893 21692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:41:05.255071 21759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:41:05.259416 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 21692
I20260620 23:41:05.259621 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781998525501163-388-0/minicluster-data/master-0/wal/instance
I20260620 23:41:05.260042 388 external_mini_cluster.cc:1428] Setting key 6a996886e066a0c8fd7d7a3794636661
I20260620 23:41:05.266918 21759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f24d5e83025b402f8691b8c9014ab169: Bootstrap starting.
I20260620 23:41:05.269196 21762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:05.262070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.182:42685 (local address 127.0.0.1:41681)
0620 23:41:05.262927 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:05.263005 (+ 78us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:05.263059 (+ 54us) server_negotiation.cc:407] Connection header received
0620 23:41:05.265375 (+ 2316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:05.265404 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:05.265814 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:05.266200 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:05.267428 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:41:05.267435 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:41:05.267449 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:41:05.267464 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:41:05.267525 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:41:05.267551 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:41:05.267738 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 23:41:05.267849 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 23:41:05.268162 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":400,"threads_started":1}
I20260620 23:41:05.270583 21759 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f24d5e83025b402f8691b8c9014ab169: Neither blocks nor log segments found. Creating new log.
I20260620 23:41:05.271823 21759 log.cc:824] T 00000000000000000000000000000000 P f24d5e83025b402f8691b8c9014ab169: Log is configured to *not* fsync() on all Append() calls
I20260620 23:41:05.273509 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:41:05.275139 21759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f24d5e83025b402f8691b8c9014ab169: No bootstrap required, opened a new log
I20260620 23:41:05.281138 21762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:05.278195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.182:41175 (local address 127.0.0.1:41681)
0620 23:41:05.278453 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:05.278467 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:05.278896 (+ 429us) server_negotiation.cc:407] Connection header received
0620 23:41:05.279145 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:05.279149 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:05.279217 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:05.279308 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:05.280303 (+ 995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:41:05.280308 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:41:05.280311 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:41:05.280331 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:41:05.280374 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:41:05.280393 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:41:05.280523 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 23:41:05.280634 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 23:41:05.280692 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 23:41:05.284528 21759 raft_consensus.cc:348] T 00000000000000000000000000000000 P f24d5e83025b402f8691b8c9014ab169 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f24d5e83025b402f8691b8c9014ab169" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41681 } }
I20260620 23:41:05.285006 21759 raft_consensus.cc:374] T 00000000000000000000000000000000 P f24d5e83025b402f8691b8c9014ab169 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:41:05.285126 21759 raft_consensus.cc:729] T 00000000000000000000000000000000 P f24d5e83025b402f8691b8c9014ab169 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f24d5e83025b402f8691b8c9014ab169, State: Initialized, Role: FOLLOWER
I20260620 23:41:05.285807 21759 consensus_queue.cc:260] T 00000000000000000000000000000000 P f24d5e83025b402f8691b8c9014ab169 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f24d5e83025b402f8691b8c9014ab169" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41681 } }
I20260620 23:41:05.286072 21759 raft_consensus.cc:388] T 00000000000000000000000000000000 P f24d5e83025b402f8691b8c9014ab169 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:41:05.286226 21759 raft_consensus.cc:482] T 00000000000000000000000000000000 P f24d5e83025b402f8691b8c9014ab169 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:41:05.286428 21759 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f24d5e83025b402f8691b8c9014ab169 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:41:05.289141 21759 raft_consensus.cc:504] T 00000000000000000000000000000000 P f24d5e83025b402f8691b8c9014ab169 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f24d5e83025b402f8691b8c9014ab169" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41681 } }
I20260620 23:41:05.289740 21759 leader_election.cc:302] T 00000000000000000000000000000000 P f24d5e83025b402f8691b8c9014ab169 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f24d5e83025b402f8691b8c9014ab169; no voters:
I20260620 23:41:05.290413 21759 leader_election.cc:288] T 00000000000000000000000000000000 P f24d5e83025b402f8691b8c9014ab169 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:41:05.290588 21769 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f24d5e83025b402f8691b8c9014ab169 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:41:05.291339 21769 raft_consensus.cc:686] T 00000000000000000000000000000000 P f24d5e83025b402f8691b8c9014ab169 [term 1 LEADER]: Becoming Leader. State: Replica: f24d5e83025b402f8691b8c9014ab169, State: Running, Role: LEADER
I20260620 23:41:05.291963 21769 consensus_queue.cc:237] T 00000000000000000000000000000000 P f24d5e83025b402f8691b8c9014ab169 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f24d5e83025b402f8691b8c9014ab169" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41681 } }
I20260620 23:41:05.292892 21759 sys_catalog.cc:564] T 00000000000000000000000000000000 P f24d5e83025b402f8691b8c9014ab169 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:41:05.295141 21771 sys_catalog.cc:455] T 00000000000000000000000000000000 P f24d5e83025b402f8691b8c9014ab169 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f24d5e83025b402f8691b8c9014ab169. Latest consensus state: current_term: 1 leader_uuid: "f24d5e83025b402f8691b8c9014ab169" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f24d5e83025b402f8691b8c9014ab169" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41681 } } }
I20260620 23:41:05.295468 21770 sys_catalog.cc:455] T 00000000000000000000000000000000 P f24d5e83025b402f8691b8c9014ab169 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f24d5e83025b402f8691b8c9014ab169" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f24d5e83025b402f8691b8c9014ab169" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41681 } } }
I20260620 23:41:05.295687 21771 sys_catalog.cc:458] T 00000000000000000000000000000000 P f24d5e83025b402f8691b8c9014ab169 [sys.catalog]: This master's current role is: LEADER
I20260620 23:41:05.295758 21770 sys_catalog.cc:458] T 00000000000000000000000000000000 P f24d5e83025b402f8691b8c9014ab169 [sys.catalog]: This master's current role is: LEADER
I20260620 23:41:05.297504 21778 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:41:05.302523 21778 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:41:05.311841 21778 catalog_manager.cc:1347] Generated new cluster ID: e9813e85e7b64252af40b3275c9362e7
I20260620 23:41:05.311980 21778 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:41:05.332271 21778 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:41:05.332823 21778 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:41:05.346659 21778 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f24d5e83025b402f8691b8c9014ab169: Generated new TSK 0
I20260620 23:41:05.347908 21778 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:41:05.358160 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 21692
W20260620 23:41:05.360302 21754 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.182:42685
W20260620 23:41:05.361116 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 21692: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:41:05.361280 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 21692
2026-06-20T23:41:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2006 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781998525501163-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:41:05 dist-test-slave-tcgx krb5kdc[21790](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:41:05 dist-test-slave-tcgx krb5kdc[21790](info): set up 2 sockets
Jun 20 23:41:05 dist-test-slave-tcgx krb5kdc[21790](info): commencing operation
krb5kdc: starting...
W20260620 23:41:06.483546 388 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781998525501163-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781998525501163-388-0/krb5kdc/test-user.keytab.
Jun 20 23:41:06 dist-test-slave-tcgx krb5kdc[21790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:41:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:41:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781998525501163-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781998525501163-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:41:06.751194 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781998525501163-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781998525501163-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781998525501163-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34481
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781998525501163-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34481
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:41:07.098100 21803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:41:07.098418 21803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:41:07.098511 21803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:41:07.105988 21803 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:41:07.106128 21803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:41:07.106175 21803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:41:07.106230 21803 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:41:07.106276 21803 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:41:07.116487 21803 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781998525501163-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781998525501163-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34481
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781998525501163-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781998525501163-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:41:07.117828 21803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:41:07.119179 21803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:41:07.131889 21808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:41:07.131929 21809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:41:07.131842 21811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:41:07.132772 21803 server_base.cc:1034] running on GCE node
I20260620 23:41:07.133594 21803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:41:07.135139 21803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:41:07.136438 21803 hybrid_clock.cc:648] HybridClock initialized: now 1781998867136368 us; error 40 us; skew 500 ppm
I20260620 23:41:07.136821 21803 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:41:07 dist-test-slave-tcgx krb5kdc[21790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998867, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:41:07.140115 21803 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:41:07.141681 21803 webserver.cc:460] Webserver started at http://127.0.0.1:45433/ using document root <none> and password file <none>
I20260620 23:41:07.142364 21803 fs_manager.cc:359] Metadata directory not provided
I20260620 23:41:07.142470 21803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:41:07.142736 21803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:41:07.145849 21803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781998525501163-388-0/minicluster-data/master-0/data/instance:
uuid: "8547bc162feb4a0a9c29ed1259918116"
format_stamp: "Formatted at 2026-06-20 23:41:07 on dist-test-slave-tcgx"
server_key: "f32a15ed926f66ba1687f7dba5df8c19"
server_key_iv: "8525926df6fea1a0db14a71d5f197bff"
server_key_version: "encryptionkey@0"
I20260620 23:41:07.146648 21803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781998525501163-388-0/minicluster-data/master-0/wal/instance:
uuid: "8547bc162feb4a0a9c29ed1259918116"
format_stamp: "Formatted at 2026-06-20 23:41:07 on dist-test-slave-tcgx"
server_key: "f32a15ed926f66ba1687f7dba5df8c19"
server_key_iv: "8525926df6fea1a0db14a71d5f197bff"
server_key_version: "encryptionkey@0"
I20260620 23:41:07.152093 21803 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 23:41:07.155781 21818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:41:07.157356 21803 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:41:07.157544 21803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781998525501163-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781998525501163-388-0/minicluster-data/master-0/wal
uuid: "8547bc162feb4a0a9c29ed1259918116"
format_stamp: "Formatted at 2026-06-20 23:41:07 on dist-test-slave-tcgx"
server_key: "f32a15ed926f66ba1687f7dba5df8c19"
server_key_iv: "8525926df6fea1a0db14a71d5f197bff"
server_key_version: "encryptionkey@0"
I20260620 23:41:07.157747 21803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781998525501163-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781998525501163-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781998525501163-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:41:07.189821 21803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:41:07.194334 21803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:41:07.194665 21803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:41:07.214031 21803 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34481
I20260620 23:41:07.214052 21869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34481 every 8 connection(s)
I20260620 23:41:07.215613 21803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:41:07.218348 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 21803
I20260620 23:41:07.218549 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781998525501163-388-0/minicluster-data/master-0/wal/instance
I20260620 23:41:07.219121 388 external_mini_cluster.cc:1428] Setting key d9003fc7b8454c903cadddf18ff5a633
I20260620 23:41:07.220237 21870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 23:41:07 dist-test-slave-tcgx krb5kdc[21790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 23:41:07.235410 21870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8547bc162feb4a0a9c29ed1259918116: Bootstrap starting.
I20260620 23:41:07.238720 21870 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8547bc162feb4a0a9c29ed1259918116: Neither blocks nor log segments found. Creating new log.
I20260620 23:41:07.239884 21870 log.cc:824] T 00000000000000000000000000000000 P 8547bc162feb4a0a9c29ed1259918116: Log is configured to *not* fsync() on all Append() calls
I20260620 23:41:07.243108 21870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8547bc162feb4a0a9c29ed1259918116: No bootstrap required, opened a new log
I20260620 23:41:07.243434 21873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:07.221086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.182:42579 (local address 127.0.0.1:34481)
0620 23:41:07.221909 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:07.221922 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:07.221963 (+ 41us) server_negotiation.cc:407] Connection header received
0620 23:41:07.223652 (+ 1689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:07.223677 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:07.224123 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:07.224480 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:07.226289 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:07.227805 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:07.229648 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:07.230317 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:07.230450 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:07.235568 (+ 5118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:41:07.235580 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:41:07.235602 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:41:07.235663 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:41:07.238916 (+ 3253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:07.239809 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:07.239816 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:07.239822 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:07.240019 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:07.240520 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:07.240524 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:07.240526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:07.241106 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:41:07.241448 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0620 23:41:07.241984 (+ 536us) server_negotiation.cc:299] Negotiation successful
0620 23:41:07.242306 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":362,"threads_started":1}
I20260620 23:41:07.247830 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:41:07.252002 21870 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8547bc162feb4a0a9c29ed1259918116 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8547bc162feb4a0a9c29ed1259918116" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34481 } }
I20260620 23:41:07.252558 21870 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8547bc162feb4a0a9c29ed1259918116 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:41:07.252707 21870 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8547bc162feb4a0a9c29ed1259918116 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8547bc162feb4a0a9c29ed1259918116, State: Initialized, Role: FOLLOWER
I20260620 23:41:07.253548 21870 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8547bc162feb4a0a9c29ed1259918116 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8547bc162feb4a0a9c29ed1259918116" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34481 } }
I20260620 23:41:07.253850 21870 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8547bc162feb4a0a9c29ed1259918116 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:41:07.254005 21870 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8547bc162feb4a0a9c29ed1259918116 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:41:07.254185 21870 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8547bc162feb4a0a9c29ed1259918116 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:41:07.257315 21870 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8547bc162feb4a0a9c29ed1259918116 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8547bc162feb4a0a9c29ed1259918116" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34481 } }
I20260620 23:41:07.257962 21870 leader_election.cc:302] T 00000000000000000000000000000000 P 8547bc162feb4a0a9c29ed1259918116 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8547bc162feb4a0a9c29ed1259918116; no voters:
I20260620 23:41:07.258744 21870 leader_election.cc:288] T 00000000000000000000000000000000 P 8547bc162feb4a0a9c29ed1259918116 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:41:07.258894 21880 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8547bc162feb4a0a9c29ed1259918116 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:41:07.259718 21880 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8547bc162feb4a0a9c29ed1259918116 [term 1 LEADER]: Becoming Leader. State: Replica: 8547bc162feb4a0a9c29ed1259918116, State: Running, Role: LEADER
I20260620 23:41:07.260438 21880 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8547bc162feb4a0a9c29ed1259918116 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8547bc162feb4a0a9c29ed1259918116" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34481 } }
I20260620 23:41:07.261801 21870 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8547bc162feb4a0a9c29ed1259918116 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:41:07.263211 21881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8547bc162feb4a0a9c29ed1259918116 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8547bc162feb4a0a9c29ed1259918116. Latest consensus state: current_term: 1 leader_uuid: "8547bc162feb4a0a9c29ed1259918116" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8547bc162feb4a0a9c29ed1259918116" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34481 } } }
I20260620 23:41:07.263716 21881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8547bc162feb4a0a9c29ed1259918116 [sys.catalog]: This master's current role is: LEADER
I20260620 23:41:07.264235 21882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8547bc162feb4a0a9c29ed1259918116 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8547bc162feb4a0a9c29ed1259918116" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8547bc162feb4a0a9c29ed1259918116" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34481 } } }
I20260620 23:41:07.264626 21882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8547bc162feb4a0a9c29ed1259918116 [sys.catalog]: This master's current role is: LEADER
I20260620 23:41:07.273237 21873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:07.252606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.182:53611 (local address 127.0.0.1:34481)
0620 23:41:07.252856 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:07.252863 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:07.253270 (+ 407us) server_negotiation.cc:407] Connection header received
0620 23:41:07.253481 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:07.253485 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:07.253554 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:07.253642 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:07.255657 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:07.256478 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:07.258318 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:07.258886 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:07.259153 (+ 267us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:07.262517 (+ 3364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:41:07.262526 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:41:07.262537 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:41:07.262597 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:41:07.267209 (+ 4612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:07.268648 (+ 1439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:07.268658 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:07.268665 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:07.268913 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:07.269663 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:07.269671 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:07.269677 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:07.270164 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:41:07.270553 (+ 389us) server_negotiation.cc:1035] Waiting for connection context
0620 23:41:07.272394 (+ 1841us) server_negotiation.cc:299] Negotiation successful
0620 23:41:07.272682 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260620 23:41:07.274297 21885 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:41:07.280727 21885 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:41:07.291817 21885 catalog_manager.cc:1347] Generated new cluster ID: 1aa9f9a972314a44b09d5971266a47a5
I20260620 23:41:07.292090 21885 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:41:07.324528 21885 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:41:07.326500 21885 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:41:07.338559 21885 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8547bc162feb4a0a9c29ed1259918116: Generated new TSK 0
I20260620 23:41:07.339708 21885 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:41:07.347011 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 21803
I20260620 23:41:07.348899 21865 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:41:07.445912 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 21803
W20260620 23:41:07.464095 21788 connection.cc:535] client connection to 127.0.0.1:34481 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34481): Connection reset by peer (error 104)
2026-06-20T23:41:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2087 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781998525501163-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:41:07 dist-test-slave-tcgx krb5kdc[21903](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:41:07 dist-test-slave-tcgx krb5kdc[21903](info): set up 2 sockets
Jun 20 23:41:07 dist-test-slave-tcgx krb5kdc[21903](info): commencing operation
krb5kdc: starting...
W20260620 23:41:08.563632 388 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781998525501163-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781998525501163-388-0/krb5kdc/test-user.keytab.
Jun 20 23:41:08 dist-test-slave-tcgx krb5kdc[21903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:41:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:41:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781998525501163-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781998525501163-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:41:08.834053 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781998525501163-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781998525501163-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781998525501163-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34049
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781998525501163-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34049
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:41:09.173448 21916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:41:09.173836 21916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:41:09.174006 21916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:41:09.181608 21916 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:41:09.181797 21916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:41:09.181912 21916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:41:09.182024 21916 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:41:09.182104 21916 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:41:09.192627 21916 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781998525501163-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781998525501163-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34049
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781998525501163-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781998525501163-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:41:09.194108 21916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:41:09.195555 21916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:41:09.208249 21921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:41:09.208225 21922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:41:09.208170 21924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:41:09.209475 21916 server_base.cc:1034] running on GCE node
I20260620 23:41:09.210367 21916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:41:09.212039 21916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:41:09.213301 21916 hybrid_clock.cc:648] HybridClock initialized: now 1781998869213254 us; error 52 us; skew 500 ppm
I20260620 23:41:09.213712 21916 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:41:09 dist-test-slave-tcgx krb5kdc[21903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998869, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:41:09.220825 21916 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:41:09.222563 21916 webserver.cc:460] Webserver started at http://127.0.0.1:35361/ using document root <none> and password file <none>
I20260620 23:41:09.223364 21916 fs_manager.cc:359] Metadata directory not provided
I20260620 23:41:09.223474 21916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:41:09.223737 21916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:41:09.226799 21916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781998525501163-388-0/minicluster-data/master-0/data/instance:
uuid: "84bf33e125c6415f9a73d19a0efd6645"
format_stamp: "Formatted at 2026-06-20 23:41:09 on dist-test-slave-tcgx"
server_key: "0e8e98f224e325240f0c2c9f4b9de779"
server_key_iv: "e1bf5f3b327b5c6b84868aaec0dc7199"
server_key_version: "encryptionkey@0"
I20260620 23:41:09.227617 21916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781998525501163-388-0/minicluster-data/master-0/wal/instance:
uuid: "84bf33e125c6415f9a73d19a0efd6645"
format_stamp: "Formatted at 2026-06-20 23:41:09 on dist-test-slave-tcgx"
server_key: "0e8e98f224e325240f0c2c9f4b9de779"
server_key_iv: "e1bf5f3b327b5c6b84868aaec0dc7199"
server_key_version: "encryptionkey@0"
I20260620 23:41:09.233512 21916 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 23:41:09.237169 21931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:41:09.238713 21916 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 23:41:09.238989 21916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781998525501163-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781998525501163-388-0/minicluster-data/master-0/wal
uuid: "84bf33e125c6415f9a73d19a0efd6645"
format_stamp: "Formatted at 2026-06-20 23:41:09 on dist-test-slave-tcgx"
server_key: "0e8e98f224e325240f0c2c9f4b9de779"
server_key_iv: "e1bf5f3b327b5c6b84868aaec0dc7199"
server_key_version: "encryptionkey@0"
I20260620 23:41:09.239228 21916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781998525501163-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781998525501163-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781998525501163-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:41:09.263125 21916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:41:09.267721 21916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:41:09.268052 21916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:41:09.287478 21916 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34049
I20260620 23:41:09.287500 21982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34049 every 8 connection(s)
I20260620 23:41:09.288975 21916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:41:09.290773 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 21916
I20260620 23:41:09.290992 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781998525501163-388-0/minicluster-data/master-0/wal/instance
I20260620 23:41:09.291558 388 external_mini_cluster.cc:1428] Setting key 24a4b2d80ec90f0e252606b561b7cd53
I20260620 23:41:09.293860 21983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:41:09.305689 21986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:09.293461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.182:33627 (local address 127.0.0.1:34049)
0620 23:41:09.294271 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:09.294283 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:09.294324 (+ 41us) server_negotiation.cc:407] Connection header received
0620 23:41:09.296063 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:09.296086 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:09.296515 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:09.296886 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:09.297527 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:09.298887 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:09.300792 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:09.301454 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:09.301590 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:09.303198 (+ 1608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:41:09.303209 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:41:09.303230 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:41:09.303854 (+ 624us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:41:09.303943 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:41:09.303971 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:41:09.304108 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 23:41:09.304430 (+ 322us) server_negotiation.cc:299] Negotiation successful
0620 23:41:09.304645 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":366,"threads_started":1}
I20260620 23:41:09.306850 21983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84bf33e125c6415f9a73d19a0efd6645: Bootstrap starting.
I20260620 23:41:09.309911 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:41:09.310523 21983 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84bf33e125c6415f9a73d19a0efd6645: Neither blocks nor log segments found. Creating new log.
I20260620 23:41:09.311645 21983 log.cc:824] T 00000000000000000000000000000000 P 84bf33e125c6415f9a73d19a0efd6645: Log is configured to *not* fsync() on all Append() calls
I20260620 23:41:09.314992 21983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84bf33e125c6415f9a73d19a0efd6645: No bootstrap required, opened a new log
I20260620 23:41:09.322053 21986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:09.314891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.182:46403 (local address 127.0.0.1:34049)
0620 23:41:09.315120 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:09.315126 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:09.315418 (+ 292us) server_negotiation.cc:407] Connection header received
0620 23:41:09.315628 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:09.315635 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:09.315699 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:09.315792 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:09.316439 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:09.317285 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:09.319239 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:09.319716 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:09.319819 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:09.321159 (+ 1340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:41:09.321165 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:41:09.321174 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:41:09.321193 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:41:09.321240 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:41:09.321256 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:41:09.321359 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 23:41:09.321605 (+ 246us) server_negotiation.cc:299] Negotiation successful
0620 23:41:09.321706 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260620 23:41:09.324810 21983 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84bf33e125c6415f9a73d19a0efd6645 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84bf33e125c6415f9a73d19a0efd6645" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34049 } }
I20260620 23:41:09.325224 21983 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84bf33e125c6415f9a73d19a0efd6645 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:41:09.325350 21983 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84bf33e125c6415f9a73d19a0efd6645 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84bf33e125c6415f9a73d19a0efd6645, State: Initialized, Role: FOLLOWER
I20260620 23:41:09.325950 21983 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84bf33e125c6415f9a73d19a0efd6645 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84bf33e125c6415f9a73d19a0efd6645" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34049 } }
I20260620 23:41:09.326205 21983 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84bf33e125c6415f9a73d19a0efd6645 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:41:09.326354 21983 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84bf33e125c6415f9a73d19a0efd6645 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:41:09.326547 21983 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84bf33e125c6415f9a73d19a0efd6645 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:41:09.328907 21983 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84bf33e125c6415f9a73d19a0efd6645 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84bf33e125c6415f9a73d19a0efd6645" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34049 } }
I20260620 23:41:09.329450 21983 leader_election.cc:302] T 00000000000000000000000000000000 P 84bf33e125c6415f9a73d19a0efd6645 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84bf33e125c6415f9a73d19a0efd6645; no voters:
I20260620 23:41:09.330119 21983 leader_election.cc:288] T 00000000000000000000000000000000 P 84bf33e125c6415f9a73d19a0efd6645 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:41:09.330286 21993 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84bf33e125c6415f9a73d19a0efd6645 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:41:09.331499 21993 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84bf33e125c6415f9a73d19a0efd6645 [term 1 LEADER]: Becoming Leader. State: Replica: 84bf33e125c6415f9a73d19a0efd6645, State: Running, Role: LEADER
I20260620 23:41:09.332034 21993 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84bf33e125c6415f9a73d19a0efd6645 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84bf33e125c6415f9a73d19a0efd6645" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34049 } }
I20260620 23:41:09.332592 21983 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84bf33e125c6415f9a73d19a0efd6645 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:41:09.335427 21994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84bf33e125c6415f9a73d19a0efd6645 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84bf33e125c6415f9a73d19a0efd6645" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84bf33e125c6415f9a73d19a0efd6645" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34049 } } }
I20260620 23:41:09.335686 21995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84bf33e125c6415f9a73d19a0efd6645 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84bf33e125c6415f9a73d19a0efd6645. Latest consensus state: current_term: 1 leader_uuid: "84bf33e125c6415f9a73d19a0efd6645" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84bf33e125c6415f9a73d19a0efd6645" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34049 } } }
I20260620 23:41:09.336009 21995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84bf33e125c6415f9a73d19a0efd6645 [sys.catalog]: This master's current role is: LEADER
I20260620 23:41:09.336014 21994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84bf33e125c6415f9a73d19a0efd6645 [sys.catalog]: This master's current role is: LEADER
I20260620 23:41:09.337651 22002 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:41:09.343756 22002 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:41:09.353183 22002 catalog_manager.cc:1347] Generated new cluster ID: 39ffb30aa5d04786b90ed3af48b236c9
I20260620 23:41:09.353367 22002 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:41:09.372416 22002 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:41:09.374450 22002 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:41:09.384960 22002 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84bf33e125c6415f9a73d19a0efd6645: Generated new TSK 0
I20260620 23:41:09.386142 22002 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:41:09.395323 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 21916
W20260620 23:41:09.396559 21978 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.182:33627
W20260620 23:41:09.397395 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 21916: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:41:09.397569 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 21916
W20260620 23:41:09.414927 21901 connection.cc:535] client connection to 127.0.0.1:34049 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34049): Connection reset by peer (error 104)
2026-06-20T23:41:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1955 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781998525501163-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:41:09 dist-test-slave-tcgx krb5kdc[22014](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:41:09 dist-test-slave-tcgx krb5kdc[22014](info): set up 2 sockets
Jun 20 23:41:09 dist-test-slave-tcgx krb5kdc[22014](info): commencing operation
krb5kdc: starting...
W20260620 23:41:10.523464 388 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781998525501163-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781998525501163-388-0/krb5kdc/test-user.keytab.
Jun 20 23:41:10 dist-test-slave-tcgx krb5kdc[22014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:41:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:41:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781998525501163-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781998525501163-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:41:10.794293 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781998525501163-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781998525501163-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781998525501163-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36201
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781998525501163-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36201
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:41:11.132968 22027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:41:11.133373 22027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:41:11.133548 22027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:41:11.141152 22027 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:41:11.141324 22027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:41:11.141440 22027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:41:11.141553 22027 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:41:11.141641 22027 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:41:11.151815 22027 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781998525501163-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781998525501163-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36201
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781998525501163-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781998525501163-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:41:11.153219 22027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:41:11.154569 22027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:41:11.166692 22033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:41:11.166749 22035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:41:11.166718 22032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:41:11.167588 22027 server_base.cc:1034] running on GCE node
I20260620 23:41:11.168395 22027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:41:11.169924 22027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:41:11.171150 22027 hybrid_clock.cc:648] HybridClock initialized: now 1781998871171099 us; error 48 us; skew 500 ppm
I20260620 23:41:11.171547 22027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:41:11 dist-test-slave-tcgx krb5kdc[22014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998871, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:41:11.174757 22027 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:41:11.176297 22027 webserver.cc:460] Webserver started at http://127.0.0.1:45959/ using document root <none> and password file <none>
I20260620 23:41:11.176960 22027 fs_manager.cc:359] Metadata directory not provided
I20260620 23:41:11.177058 22027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:41:11.177333 22027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:41:11.180629 22027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781998525501163-388-0/minicluster-data/master-0/data/instance:
uuid: "4b873be20cea4775abe27e70357135bb"
format_stamp: "Formatted at 2026-06-20 23:41:11 on dist-test-slave-tcgx"
server_key: "f6648064586ec7338becb703d25b4a4b"
server_key_iv: "2cebf3b57666bc5cb3e5db675e28448e"
server_key_version: "encryptionkey@0"
I20260620 23:41:11.181388 22027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781998525501163-388-0/minicluster-data/master-0/wal/instance:
uuid: "4b873be20cea4775abe27e70357135bb"
format_stamp: "Formatted at 2026-06-20 23:41:11 on dist-test-slave-tcgx"
server_key: "f6648064586ec7338becb703d25b4a4b"
server_key_iv: "2cebf3b57666bc5cb3e5db675e28448e"
server_key_version: "encryptionkey@0"
I20260620 23:41:11.186926 22027 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 23:41:11.190635 22042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:41:11.192188 22027 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:41:11.192442 22027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781998525501163-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781998525501163-388-0/minicluster-data/master-0/wal
uuid: "4b873be20cea4775abe27e70357135bb"
format_stamp: "Formatted at 2026-06-20 23:41:11 on dist-test-slave-tcgx"
server_key: "f6648064586ec7338becb703d25b4a4b"
server_key_iv: "2cebf3b57666bc5cb3e5db675e28448e"
server_key_version: "encryptionkey@0"
I20260620 23:41:11.192662 22027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781998525501163-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781998525501163-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781998525501163-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:41:11.198670 22027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:41:11.203306 22027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:41:11.203609 22027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:41:11.222759 22027 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36201
I20260620 23:41:11.222800 22093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36201 every 8 connection(s)
I20260620 23:41:11.224263 22027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:41:11.228250 22094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:41:11.230309 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 22027
I20260620 23:41:11.230516 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781998525501163-388-0/minicluster-data/master-0/wal/instance
I20260620 23:41:11.231069 388 external_mini_cluster.cc:1428] Setting key dc4eaa4e7244ed19a1c69d29f8716061
I20260620 23:41:11.239203 22097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:11.233034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.182:46443 (local address 127.0.0.1:36201)
0620 23:41:11.233858 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:11.233874 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:11.233921 (+ 47us) server_negotiation.cc:407] Connection header received
0620 23:41:11.235601 (+ 1680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:11.235628 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:11.236036 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:11.236464 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:11.237492 (+ 1028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:41:11.237499 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:41:11.237512 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:41:11.237527 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:41:11.237583 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:41:11.237602 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:41:11.237865 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0620 23:41:11.238009 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 23:41:11.238195 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":340,"threads_started":1}
I20260620 23:41:11.243078 22094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b873be20cea4775abe27e70357135bb: Bootstrap starting.
I20260620 23:41:11.243151 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:41:11.246378 22094 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b873be20cea4775abe27e70357135bb: Neither blocks nor log segments found. Creating new log.
I20260620 23:41:11.247658 22094 log.cc:824] T 00000000000000000000000000000000 P 4b873be20cea4775abe27e70357135bb: Log is configured to *not* fsync() on all Append() calls
I20260620 23:41:11.250142 22097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:11.247668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.182:52179 (local address 127.0.0.1:36201)
0620 23:41:11.247866 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:11.247872 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:11.248099 (+ 227us) server_negotiation.cc:407] Connection header received
0620 23:41:11.248291 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:11.248297 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:11.248365 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:11.248486 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:11.249468 (+ 982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:41:11.249472 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:41:11.249475 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:41:11.249489 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:41:11.249527 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:41:11.249542 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:41:11.249635 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 23:41:11.249785 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 23:41:11.249835 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 23:41:11.251160 22094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b873be20cea4775abe27e70357135bb: No bootstrap required, opened a new log
I20260620 23:41:11.258591 22094 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b873be20cea4775abe27e70357135bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b873be20cea4775abe27e70357135bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36201 } }
I20260620 23:41:11.259065 22094 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b873be20cea4775abe27e70357135bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:41:11.259152 22094 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b873be20cea4775abe27e70357135bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b873be20cea4775abe27e70357135bb, State: Initialized, Role: FOLLOWER
I20260620 23:41:11.259691 22094 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b873be20cea4775abe27e70357135bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b873be20cea4775abe27e70357135bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36201 } }
I20260620 23:41:11.259903 22094 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b873be20cea4775abe27e70357135bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:41:11.260004 22094 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b873be20cea4775abe27e70357135bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:41:11.260129 22094 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b873be20cea4775abe27e70357135bb [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:41:11.262305 22094 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b873be20cea4775abe27e70357135bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b873be20cea4775abe27e70357135bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36201 } }
I20260620 23:41:11.262782 22094 leader_election.cc:302] T 00000000000000000000000000000000 P 4b873be20cea4775abe27e70357135bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b873be20cea4775abe27e70357135bb; no voters:
I20260620 23:41:11.263475 22094 leader_election.cc:288] T 00000000000000000000000000000000 P 4b873be20cea4775abe27e70357135bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:41:11.263576 22104 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b873be20cea4775abe27e70357135bb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:41:11.264319 22104 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b873be20cea4775abe27e70357135bb [term 1 LEADER]: Becoming Leader. State: Replica: 4b873be20cea4775abe27e70357135bb, State: Running, Role: LEADER
I20260620 23:41:11.264966 22104 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b873be20cea4775abe27e70357135bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b873be20cea4775abe27e70357135bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36201 } }
I20260620 23:41:11.265821 22094 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b873be20cea4775abe27e70357135bb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:41:11.268051 22105 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b873be20cea4775abe27e70357135bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b873be20cea4775abe27e70357135bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b873be20cea4775abe27e70357135bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36201 } } }
I20260620 23:41:11.268555 22105 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b873be20cea4775abe27e70357135bb [sys.catalog]: This master's current role is: LEADER
I20260620 23:41:11.268774 22106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b873be20cea4775abe27e70357135bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b873be20cea4775abe27e70357135bb. Latest consensus state: current_term: 1 leader_uuid: "4b873be20cea4775abe27e70357135bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b873be20cea4775abe27e70357135bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36201 } } }
I20260620 23:41:11.269217 22106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b873be20cea4775abe27e70357135bb [sys.catalog]: This master's current role is: LEADER
I20260620 23:41:11.270780 22111 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:41:11.276398 22111 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:41:11.285638 22111 catalog_manager.cc:1347] Generated new cluster ID: 4e13f5c97cce49b8b563987bb083bded
I20260620 23:41:11.285779 22111 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:41:11.306181 22111 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:41:11.306625 22111 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:41:11.315987 22111 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b873be20cea4775abe27e70357135bb: Generated new TSK 0
I20260620 23:41:11.316880 22111 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:41:11.323676 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 22027
W20260620 23:41:11.324836 22089 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.182:46443
W20260620 23:41:11.325543 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 22027: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:41:11.325688 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 22027
2026-06-20T23:41:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1920 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781998525501163-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:41:11 dist-test-slave-tcgx krb5kdc[22125](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:41:11 dist-test-slave-tcgx krb5kdc[22125](info): set up 2 sockets
Jun 20 23:41:11 dist-test-slave-tcgx krb5kdc[22125](info): commencing operation
krb5kdc: starting...
W20260620 23:41:12.435497 388 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.006s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781998525501163-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781998525501163-388-0/krb5kdc/test-user.keytab.
Jun 20 23:41:12 dist-test-slave-tcgx krb5kdc[22125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:41:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:41:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 23:41:12.706758 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781998525501163-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781998525501163-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781998525501163-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40613
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40613
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:41:13.050509 22138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:41:13.050890 22138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:41:13.051095 22138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:41:13.058651 22138 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:41:13.058822 22138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:41:13.058924 22138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:41:13.059074 22138 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:41:13.059156 22138 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:41:13.069346 22138 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781998525501163-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781998525501163-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40613
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781998525501163-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:41:13.070822 22138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:41:13.072249 22138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:41:13.085040 22146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:41:13.085057 22144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:41:13.085117 22143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:41:13.085886 22138 server_base.cc:1034] running on GCE node
I20260620 23:41:13.086791 22138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:41:13.088454 22138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:41:13.089758 22138 hybrid_clock.cc:648] HybridClock initialized: now 1781998873089692 us; error 55 us; skew 500 ppm
I20260620 23:41:13.090195 22138 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:41:13 dist-test-slave-tcgx krb5kdc[22125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998873, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:41:13.093528 22138 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 23:41:13.095043 22138 webserver.cc:460] Webserver started at http://127.0.97.62:44079/ using document root <none> and password file <none>
I20260620 23:41:13.095729 22138 fs_manager.cc:359] Metadata directory not provided
I20260620 23:41:13.095841 22138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:41:13.096104 22138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:41:13.099277 22138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781998525501163-388-0/minicluster-data/master-0/data/instance:
uuid: "45bf8f04ce2f48a9ad48cf0c276f70e1"
format_stamp: "Formatted at 2026-06-20 23:41:13 on dist-test-slave-tcgx"
server_key: "c25fab623413154485dfa1051150f462"
server_key_iv: "867b93f1e075addb8e858514d8e94f65"
server_key_version: "encryptionkey@0"
I20260620 23:41:13.100010 22138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781998525501163-388-0/minicluster-data/master-0/wal/instance:
uuid: "45bf8f04ce2f48a9ad48cf0c276f70e1"
format_stamp: "Formatted at 2026-06-20 23:41:13 on dist-test-slave-tcgx"
server_key: "c25fab623413154485dfa1051150f462"
server_key_iv: "867b93f1e075addb8e858514d8e94f65"
server_key_version: "encryptionkey@0"
I20260620 23:41:13.105453 22138 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 23:41:13.109263 22153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:41:13.110857 22138 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:41:13.111136 22138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781998525501163-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781998525501163-388-0/minicluster-data/master-0/wal
uuid: "45bf8f04ce2f48a9ad48cf0c276f70e1"
format_stamp: "Formatted at 2026-06-20 23:41:13 on dist-test-slave-tcgx"
server_key: "c25fab623413154485dfa1051150f462"
server_key_iv: "867b93f1e075addb8e858514d8e94f65"
server_key_version: "encryptionkey@0"
I20260620 23:41:13.111379 22138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781998525501163-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781998525501163-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781998525501163-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:41:13.153759 22138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:41:13.158454 22138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:41:13.158703 22138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:41:13.177289 22138 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40613
I20260620 23:41:13.177309 22204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40613 every 8 connection(s)
I20260620 23:41:13.178740 22138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:41:13.182706 22205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:41:13.183180 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 22138
I20260620 23:41:13.183352 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781998525501163-388-0/minicluster-data/master-0/wal/instance
I20260620 23:41:13.183790 388 external_mini_cluster.cc:1428] Setting key e87581481e393f6eaff58b2f3b7ade48
I20260620 23:41:13.198046 22205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45bf8f04ce2f48a9ad48cf0c276f70e1: Bootstrap starting.
Jun 20 23:41:13 dist-test-slave-tcgx krb5kdc[22125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:41:13.201547 22205 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45bf8f04ce2f48a9ad48cf0c276f70e1: Neither blocks nor log segments found. Creating new log.
I20260620 23:41:13.203056 22205 log.cc:824] T 00000000000000000000000000000000 P 45bf8f04ce2f48a9ad48cf0c276f70e1: Log is configured to *not* fsync() on all Append() calls
I20260620 23:41:13.206691 22205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45bf8f04ce2f48a9ad48cf0c276f70e1: No bootstrap required, opened a new log
I20260620 23:41:13.210510 22208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:13.185712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56687 (local address 127.0.97.62:40613)
0620 23:41:13.186628 (+ 916us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:13.186644 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:13.186700 (+ 56us) server_negotiation.cc:407] Connection header received
0620 23:41:13.189632 (+ 2932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:13.189665 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:13.190268 (+ 603us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:13.190779 (+ 511us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:13.192197 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:13.194270 (+ 2073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:13.195775 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:13.196897 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:13.197084 (+ 187us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:13.201449 (+ 4365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:41:13.201463 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:41:13.201485 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:41:13.201551 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:41:13.205509 (+ 3958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:13.206346 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:13.206355 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:13.206363 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:13.206590 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:13.207132 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:13.207137 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:13.207140 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:13.207916 (+ 776us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:41:13.208271 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0620 23:41:13.208618 (+ 347us) server_negotiation.cc:299] Negotiation successful
0620 23:41:13.209015 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":411,"threads_started":1}
I20260620 23:41:13.214859 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:41:13.216884 22205 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45bf8f04ce2f48a9ad48cf0c276f70e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45bf8f04ce2f48a9ad48cf0c276f70e1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40613 } }
I20260620 23:41:13.217295 22205 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45bf8f04ce2f48a9ad48cf0c276f70e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:41:13.217433 22205 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45bf8f04ce2f48a9ad48cf0c276f70e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45bf8f04ce2f48a9ad48cf0c276f70e1, State: Initialized, Role: FOLLOWER
I20260620 23:41:13.218144 22205 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45bf8f04ce2f48a9ad48cf0c276f70e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45bf8f04ce2f48a9ad48cf0c276f70e1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40613 } }
I20260620 23:41:13.218417 22205 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45bf8f04ce2f48a9ad48cf0c276f70e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:41:13.218578 22205 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45bf8f04ce2f48a9ad48cf0c276f70e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:41:13.218771 22205 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45bf8f04ce2f48a9ad48cf0c276f70e1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:41:13.221670 22205 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45bf8f04ce2f48a9ad48cf0c276f70e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45bf8f04ce2f48a9ad48cf0c276f70e1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40613 } }
I20260620 23:41:13.222267 22205 leader_election.cc:302] T 00000000000000000000000000000000 P 45bf8f04ce2f48a9ad48cf0c276f70e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45bf8f04ce2f48a9ad48cf0c276f70e1; no voters:
I20260620 23:41:13.222995 22205 leader_election.cc:288] T 00000000000000000000000000000000 P 45bf8f04ce2f48a9ad48cf0c276f70e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:41:13.223162 22215 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45bf8f04ce2f48a9ad48cf0c276f70e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:41:13.223827 22215 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45bf8f04ce2f48a9ad48cf0c276f70e1 [term 1 LEADER]: Becoming Leader. State: Replica: 45bf8f04ce2f48a9ad48cf0c276f70e1, State: Running, Role: LEADER
I20260620 23:41:13.224481 22215 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45bf8f04ce2f48a9ad48cf0c276f70e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45bf8f04ce2f48a9ad48cf0c276f70e1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40613 } }
I20260620 23:41:13.226284 22215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45bf8f04ce2f48a9ad48cf0c276f70e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45bf8f04ce2f48a9ad48cf0c276f70e1. Latest consensus state: current_term: 1 leader_uuid: "45bf8f04ce2f48a9ad48cf0c276f70e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45bf8f04ce2f48a9ad48cf0c276f70e1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40613 } } }
I20260620 23:41:13.226622 22215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45bf8f04ce2f48a9ad48cf0c276f70e1 [sys.catalog]: This master's current role is: LEADER
I20260620 23:41:13.227177 22217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45bf8f04ce2f48a9ad48cf0c276f70e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45bf8f04ce2f48a9ad48cf0c276f70e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45bf8f04ce2f48a9ad48cf0c276f70e1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40613 } } }
I20260620 23:41:13.227566 22217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45bf8f04ce2f48a9ad48cf0c276f70e1 [sys.catalog]: This master's current role is: LEADER
I20260620 23:41:13.230561 22205 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45bf8f04ce2f48a9ad48cf0c276f70e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:41:13.234210 22219 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:41:13.240477 22208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:13.219859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41613 (local address 127.0.97.62:40613)
0620 23:41:13.220052 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:13.220059 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:13.220412 (+ 353us) server_negotiation.cc:407] Connection header received
0620 23:41:13.220546 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:13.220553 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:13.220641 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:13.220755 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:13.222207 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:13.223522 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:13.226131 (+ 2609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:13.227146 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:13.227318 (+ 172us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:13.231159 (+ 3841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:41:13.231169 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:41:13.231177 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:41:13.231242 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:41:13.235974 (+ 4732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:13.236829 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:13.236835 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:13.236839 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:13.237022 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:13.237689 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:13.237693 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:13.237696 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:13.238124 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:41:13.239681 (+ 1557us) server_negotiation.cc:1035] Waiting for connection context
0620 23:41:13.239887 (+ 206us) server_negotiation.cc:299] Negotiation successful
0620 23:41:13.240098 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 23:41:13.243273 22219 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:41:13.254009 22219 catalog_manager.cc:1347] Generated new cluster ID: 0ca90bbdbeb34449ac4848e32bf52d91
I20260620 23:41:13.254144 22219 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:41:13.263352 22219 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:41:13.264757 22219 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:41:13.274825 22219 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45bf8f04ce2f48a9ad48cf0c276f70e1: Generated new TSK 0
I20260620 23:41:13.275908 22219 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:41:13.283097 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 22138
I20260620 23:41:13.284469 22200 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:41:13.376430 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 22138
W20260620 23:41:13.394263 22123 connection.cc:535] client connection to 127.0.97.62:40613 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:40613): Connection reset by peer (error 104)
2026-06-20T23:41:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2054 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781998525501163-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:41:13 dist-test-slave-tcgx krb5kdc[22237](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:41:13 dist-test-slave-tcgx krb5kdc[22237](info): set up 2 sockets
Jun 20 23:41:13 dist-test-slave-tcgx krb5kdc[22237](info): commencing operation
krb5kdc: starting...
W20260620 23:41:14.547508 388 mini_kdc.cc:121] Time spent starting KDC: real 1.134s user 0.013s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781998525501163-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781998525501163-388-0/krb5kdc/test-user.keytab.
Jun 20 23:41:14 dist-test-slave-tcgx krb5kdc[22237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:41:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:41:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 23:41:14.815521 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781998525501163-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781998525501163-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781998525501163-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36333
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36333
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:41:15.157001 22250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:41:15.157313 22250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:41:15.157487 22250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:41:15.164888 22250 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:41:15.165022 22250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:41:15.165118 22250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:41:15.165235 22250 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:41:15.165318 22250 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:41:15.175460 22250 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781998525501163-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781998525501163-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36333
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781998525501163-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:41:15.176843 22250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:41:15.178220 22250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:41:15.190383 22258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:41:15.190558 22255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:41:15.190438 22256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:41:15.191700 22250 server_base.cc:1034] running on GCE node
I20260620 23:41:15.192580 22250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:41:15.194226 22250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:41:15.195470 22250 hybrid_clock.cc:648] HybridClock initialized: now 1781998875195408 us; error 58 us; skew 500 ppm
I20260620 23:41:15.195847 22250 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:41:15 dist-test-slave-tcgx krb5kdc[22237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:41:15.204069 22250 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 23:41:15.205705 22250 webserver.cc:460] Webserver started at http://127.0.97.62:45065/ using document root <none> and password file <none>
I20260620 23:41:15.206466 22250 fs_manager.cc:359] Metadata directory not provided
I20260620 23:41:15.206622 22250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:41:15.207005 22250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:41:15.210228 22250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781998525501163-388-0/minicluster-data/master-0/data/instance:
uuid: "7a6deeaf31ee4b669bbfad69f93752ad"
format_stamp: "Formatted at 2026-06-20 23:41:15 on dist-test-slave-tcgx"
server_key: "d6d04f00da84b78549a5191ad5805e29"
server_key_iv: "bf50a690066751fadd754eb895e7b18c"
server_key_version: "encryptionkey@0"
I20260620 23:41:15.211115 22250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781998525501163-388-0/minicluster-data/master-0/wal/instance:
uuid: "7a6deeaf31ee4b669bbfad69f93752ad"
format_stamp: "Formatted at 2026-06-20 23:41:15 on dist-test-slave-tcgx"
server_key: "d6d04f00da84b78549a5191ad5805e29"
server_key_iv: "bf50a690066751fadd754eb895e7b18c"
server_key_version: "encryptionkey@0"
I20260620 23:41:15.216753 22250 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 23:41:15.220508 22265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:41:15.221885 22250 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:41:15.222090 22250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781998525501163-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781998525501163-388-0/minicluster-data/master-0/wal
uuid: "7a6deeaf31ee4b669bbfad69f93752ad"
format_stamp: "Formatted at 2026-06-20 23:41:15 on dist-test-slave-tcgx"
server_key: "d6d04f00da84b78549a5191ad5805e29"
server_key_iv: "bf50a690066751fadd754eb895e7b18c"
server_key_version: "encryptionkey@0"
I20260620 23:41:15.222309 22250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781998525501163-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781998525501163-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781998525501163-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:41:15.236353 22250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:41:15.241299 22250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:41:15.241631 22250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:41:15.260774 22250 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36333
I20260620 23:41:15.260803 22316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36333 every 8 connection(s)
I20260620 23:41:15.262324 22250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:41:15.263512 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 22250
I20260620 23:41:15.263706 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781998525501163-388-0/minicluster-data/master-0/wal/instance
I20260620 23:41:15.264236 388 external_mini_cluster.cc:1428] Setting key fcfa652af0ae9daf638f3330ffaa7403
I20260620 23:41:15.267503 22317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:41:15.278105 22320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:15.266194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48065 (local address 127.0.97.62:36333)
0620 23:41:15.267083 (+ 889us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:15.267095 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:15.267138 (+ 43us) server_negotiation.cc:407] Connection header received
0620 23:41:15.268880 (+ 1742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:15.268903 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:15.269335 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:15.269769 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:15.270403 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:15.271808 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:15.273192 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:15.273964 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:15.274164 (+ 200us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:15.275549 (+ 1385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:41:15.275557 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:41:15.275577 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:41:15.276192 (+ 615us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:41:15.276301 (+ 109us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:41:15.276331 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:41:15.276523 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0620 23:41:15.276874 (+ 351us) server_negotiation.cc:299] Negotiation successful
0620 23:41:15.277066 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":340,"threads_started":1}
I20260620 23:41:15.279001 22317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a6deeaf31ee4b669bbfad69f93752ad: Bootstrap starting.
I20260620 23:41:15.282397 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:41:15.282682 22317 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a6deeaf31ee4b669bbfad69f93752ad: Neither blocks nor log segments found. Creating new log.
I20260620 23:41:15.283787 22317 log.cc:824] T 00000000000000000000000000000000 P 7a6deeaf31ee4b669bbfad69f93752ad: Log is configured to *not* fsync() on all Append() calls
I20260620 23:41:15.287030 22317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a6deeaf31ee4b669bbfad69f93752ad: No bootstrap required, opened a new log
I20260620 23:41:15.293982 22320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:15.287346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42121 (local address 127.0.97.62:36333)
0620 23:41:15.287511 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:15.287516 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:15.288022 (+ 506us) server_negotiation.cc:407] Connection header received
0620 23:41:15.288283 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:15.288287 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:15.288349 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:15.288455 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:15.288990 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:15.289802 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:15.291262 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:15.291755 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:15.291945 (+ 190us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:15.293194 (+ 1249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:41:15.293199 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:41:15.293203 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:41:15.293223 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:41:15.293263 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:41:15.293279 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:41:15.293411 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 23:41:15.293612 (+ 201us) server_negotiation.cc:299] Negotiation successful
0620 23:41:15.293675 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 23:41:15.295892 22317 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a6deeaf31ee4b669bbfad69f93752ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a6deeaf31ee4b669bbfad69f93752ad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36333 } }
I20260620 23:41:15.296361 22317 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a6deeaf31ee4b669bbfad69f93752ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:41:15.296476 22317 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a6deeaf31ee4b669bbfad69f93752ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a6deeaf31ee4b669bbfad69f93752ad, State: Initialized, Role: FOLLOWER
I20260620 23:41:15.297158 22317 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a6deeaf31ee4b669bbfad69f93752ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a6deeaf31ee4b669bbfad69f93752ad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36333 } }
I20260620 23:41:15.297425 22317 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a6deeaf31ee4b669bbfad69f93752ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:41:15.297578 22317 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a6deeaf31ee4b669bbfad69f93752ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:41:15.297760 22317 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a6deeaf31ee4b669bbfad69f93752ad [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:41:15.300484 22317 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a6deeaf31ee4b669bbfad69f93752ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a6deeaf31ee4b669bbfad69f93752ad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36333 } }
I20260620 23:41:15.301039 22317 leader_election.cc:302] T 00000000000000000000000000000000 P 7a6deeaf31ee4b669bbfad69f93752ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a6deeaf31ee4b669bbfad69f93752ad; no voters:
I20260620 23:41:15.301690 22317 leader_election.cc:288] T 00000000000000000000000000000000 P 7a6deeaf31ee4b669bbfad69f93752ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:41:15.301837 22327 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a6deeaf31ee4b669bbfad69f93752ad [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:41:15.302582 22327 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a6deeaf31ee4b669bbfad69f93752ad [term 1 LEADER]: Becoming Leader. State: Replica: 7a6deeaf31ee4b669bbfad69f93752ad, State: Running, Role: LEADER
I20260620 23:41:15.303198 22327 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a6deeaf31ee4b669bbfad69f93752ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a6deeaf31ee4b669bbfad69f93752ad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36333 } }
I20260620 23:41:15.303964 22317 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a6deeaf31ee4b669bbfad69f93752ad [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:41:15.306180 22329 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a6deeaf31ee4b669bbfad69f93752ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a6deeaf31ee4b669bbfad69f93752ad. Latest consensus state: current_term: 1 leader_uuid: "7a6deeaf31ee4b669bbfad69f93752ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a6deeaf31ee4b669bbfad69f93752ad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36333 } } }
I20260620 23:41:15.306463 22328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a6deeaf31ee4b669bbfad69f93752ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a6deeaf31ee4b669bbfad69f93752ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a6deeaf31ee4b669bbfad69f93752ad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36333 } } }
I20260620 23:41:15.306720 22329 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a6deeaf31ee4b669bbfad69f93752ad [sys.catalog]: This master's current role is: LEADER
I20260620 23:41:15.306774 22328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a6deeaf31ee4b669bbfad69f93752ad [sys.catalog]: This master's current role is: LEADER
I20260620 23:41:15.307885 22333 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:41:15.315042 22333 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:41:15.324601 22333 catalog_manager.cc:1347] Generated new cluster ID: 22c668a8840243d8a181d3bef7db31e3
I20260620 23:41:15.324735 22333 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:41:15.354586 22333 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:41:15.356634 22333 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:41:15.367070 22333 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a6deeaf31ee4b669bbfad69f93752ad: Generated new TSK 0
I20260620 23:41:15.368213 22333 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:41:15.381035 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 22250
W20260620 23:41:15.382216 22312 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:48065
W20260620 23:41:15.383059 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 22250: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:41:15.383226 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 22250
W20260620 23:41:15.400576 22235 connection.cc:535] client connection to 127.0.97.62:36333 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:36333): Connection reset by peer (error 104)
2026-06-20T23:41:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2005 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781998525501163-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:41:15 dist-test-slave-tcgx krb5kdc[22348](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:41:15 dist-test-slave-tcgx krb5kdc[22348](info): set up 2 sockets
Jun 20 23:41:15 dist-test-slave-tcgx krb5kdc[22348](info): commencing operation
krb5kdc: starting...
W20260620 23:41:16.551504 388 mini_kdc.cc:121] Time spent starting KDC: real 1.132s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781998525501163-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781998525501163-388-0/krb5kdc/test-user.keytab.
Jun 20 23:41:16 dist-test-slave-tcgx krb5kdc[22348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:41:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:41:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 23:41:16.818197 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781998525501163-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781998525501163-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781998525501163-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35935
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35935
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:41:17.161820 22361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:41:17.162142 22361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:41:17.162292 22361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:41:17.169657 22361 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:41:17.169795 22361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:41:17.169842 22361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:41:17.169895 22361 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:41:17.169945 22361 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:41:17.180001 22361 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781998525501163-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781998525501163-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35935
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781998525501163-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:41:17.181306 22361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:41:17.182600 22361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:41:17.194854 22366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:41:17.194805 22369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:41:17.194907 22367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:41:17.195827 22361 server_base.cc:1034] running on GCE node
I20260620 23:41:17.196666 22361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:41:17.198129 22361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:41:17.199370 22361 hybrid_clock.cc:648] HybridClock initialized: now 1781998877199337 us; error 47 us; skew 500 ppm
I20260620 23:41:17.199736 22361 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:41:17 dist-test-slave-tcgx krb5kdc[22348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998877, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:41:17.202867 22361 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 23:41:17.204442 22361 webserver.cc:460] Webserver started at http://127.0.97.62:46397/ using document root <none> and password file <none>
I20260620 23:41:17.205161 22361 fs_manager.cc:359] Metadata directory not provided
I20260620 23:41:17.205317 22361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:41:17.205647 22361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:41:17.208933 22361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781998525501163-388-0/minicluster-data/master-0/data/instance:
uuid: "b9c11e255a7e4b16988638c906ea01ae"
format_stamp: "Formatted at 2026-06-20 23:41:17 on dist-test-slave-tcgx"
server_key: "a3d6d72b48f4389c63e584aec7574c24"
server_key_iv: "3cefc70f1c6c7f75682a1d862547ed19"
server_key_version: "encryptionkey@0"
I20260620 23:41:17.209745 22361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781998525501163-388-0/minicluster-data/master-0/wal/instance:
uuid: "b9c11e255a7e4b16988638c906ea01ae"
format_stamp: "Formatted at 2026-06-20 23:41:17 on dist-test-slave-tcgx"
server_key: "a3d6d72b48f4389c63e584aec7574c24"
server_key_iv: "3cefc70f1c6c7f75682a1d862547ed19"
server_key_version: "encryptionkey@0"
I20260620 23:41:17.215237 22361 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 23:41:17.219026 22376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:41:17.220530 22361 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:41:17.220716 22361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781998525501163-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781998525501163-388-0/minicluster-data/master-0/wal
uuid: "b9c11e255a7e4b16988638c906ea01ae"
format_stamp: "Formatted at 2026-06-20 23:41:17 on dist-test-slave-tcgx"
server_key: "a3d6d72b48f4389c63e584aec7574c24"
server_key_iv: "3cefc70f1c6c7f75682a1d862547ed19"
server_key_version: "encryptionkey@0"
I20260620 23:41:17.220943 22361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781998525501163-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781998525501163-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781998525501163-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:41:17.226869 22361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:41:17.231117 22361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:41:17.231449 22361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:41:17.250142 22361 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35935
I20260620 23:41:17.250187 22427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35935 every 8 connection(s)
I20260620 23:41:17.251732 22361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:41:17.254336 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 22361
I20260620 23:41:17.254534 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781998525501163-388-0/minicluster-data/master-0/wal/instance
I20260620 23:41:17.255082 388 external_mini_cluster.cc:1428] Setting key 89fcfd0162de12b649cfae84ed7d660e
I20260620 23:41:17.256172 22428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:41:17.263756 22431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:17.256999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44493 (local address 127.0.97.62:35935)
0620 23:41:17.258026 (+ 1027us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:17.258051 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:17.258313 (+ 262us) server_negotiation.cc:407] Connection header received
0620 23:41:17.260072 (+ 1759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:17.260094 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:17.260511 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:17.260856 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:17.261925 (+ 1069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:41:17.261931 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:41:17.261944 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:41:17.261964 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:41:17.262044 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:41:17.262072 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:41:17.262298 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0620 23:41:17.262408 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 23:41:17.262659 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":724,"thread_start_us":399,"threads_started":1}
I20260620 23:41:17.267668 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:41:17.270069 22428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9c11e255a7e4b16988638c906ea01ae: Bootstrap starting.
I20260620 23:41:17.274051 22428 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9c11e255a7e4b16988638c906ea01ae: Neither blocks nor log segments found. Creating new log.
I20260620 23:41:17.274875 22431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:17.272141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33811 (local address 127.0.97.62:35935)
0620 23:41:17.272406 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:17.272416 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:17.272720 (+ 304us) server_negotiation.cc:407] Connection header received
0620 23:41:17.272954 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:17.272958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:17.273020 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:17.273128 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:17.274134 (+ 1006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:41:17.274141 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:41:17.274149 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:41:17.274166 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:41:17.274262 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:41:17.274288 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:41:17.274413 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 23:41:17.274545 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 23:41:17.274607 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260620 23:41:17.275399 22428 log.cc:824] T 00000000000000000000000000000000 P b9c11e255a7e4b16988638c906ea01ae: Log is configured to *not* fsync() on all Append() calls
I20260620 23:41:17.278649 22428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9c11e255a7e4b16988638c906ea01ae: No bootstrap required, opened a new log
I20260620 23:41:17.286392 22428 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9c11e255a7e4b16988638c906ea01ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9c11e255a7e4b16988638c906ea01ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35935 } }
I20260620 23:41:17.286832 22428 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9c11e255a7e4b16988638c906ea01ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:41:17.286970 22428 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9c11e255a7e4b16988638c906ea01ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9c11e255a7e4b16988638c906ea01ae, State: Initialized, Role: FOLLOWER
I20260620 23:41:17.287575 22428 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9c11e255a7e4b16988638c906ea01ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9c11e255a7e4b16988638c906ea01ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35935 } }
I20260620 23:41:17.287827 22428 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9c11e255a7e4b16988638c906ea01ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:41:17.287981 22428 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9c11e255a7e4b16988638c906ea01ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:41:17.288157 22428 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9c11e255a7e4b16988638c906ea01ae [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:41:17.290705 22428 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9c11e255a7e4b16988638c906ea01ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9c11e255a7e4b16988638c906ea01ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35935 } }
I20260620 23:41:17.291280 22428 leader_election.cc:302] T 00000000000000000000000000000000 P b9c11e255a7e4b16988638c906ea01ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9c11e255a7e4b16988638c906ea01ae; no voters:
I20260620 23:41:17.291930 22428 leader_election.cc:288] T 00000000000000000000000000000000 P b9c11e255a7e4b16988638c906ea01ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:41:17.292073 22438 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9c11e255a7e4b16988638c906ea01ae [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:41:17.292795 22438 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9c11e255a7e4b16988638c906ea01ae [term 1 LEADER]: Becoming Leader. State: Replica: b9c11e255a7e4b16988638c906ea01ae, State: Running, Role: LEADER
I20260620 23:41:17.293326 22438 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9c11e255a7e4b16988638c906ea01ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9c11e255a7e4b16988638c906ea01ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35935 } }
I20260620 23:41:17.294086 22428 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9c11e255a7e4b16988638c906ea01ae [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:41:17.296360 22440 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9c11e255a7e4b16988638c906ea01ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9c11e255a7e4b16988638c906ea01ae. Latest consensus state: current_term: 1 leader_uuid: "b9c11e255a7e4b16988638c906ea01ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9c11e255a7e4b16988638c906ea01ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35935 } } }
I20260620 23:41:17.296583 22439 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9c11e255a7e4b16988638c906ea01ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9c11e255a7e4b16988638c906ea01ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9c11e255a7e4b16988638c906ea01ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35935 } } }
I20260620 23:41:17.296890 22439 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9c11e255a7e4b16988638c906ea01ae [sys.catalog]: This master's current role is: LEADER
I20260620 23:41:17.296887 22440 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9c11e255a7e4b16988638c906ea01ae [sys.catalog]: This master's current role is: LEADER
I20260620 23:41:17.297855 22447 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:41:17.303825 22447 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:41:17.312971 22447 catalog_manager.cc:1347] Generated new cluster ID: b3032479d28446dfa336b6e63e602604
I20260620 23:41:17.313118 22447 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:41:17.327972 22447 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:41:17.328509 22447 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:41:17.341003 22447 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9c11e255a7e4b16988638c906ea01ae: Generated new TSK 0
I20260620 23:41:17.342152 22447 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:41:17.350634 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 22361
W20260620 23:41:17.351938 22423 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:44493
W20260620 23:41:17.352631 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 22361: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:41:17.352797 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 22361
2026-06-20T23:41:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1969 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781998525501163-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:41:17 dist-test-slave-tcgx krb5kdc[22459](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:41:17 dist-test-slave-tcgx krb5kdc[22459](info): set up 2 sockets
Jun 20 23:41:17 dist-test-slave-tcgx krb5kdc[22459](info): commencing operation
krb5kdc: starting...
W20260620 23:41:18.463587 388 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781998525501163-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781998525501163-388-0/krb5kdc/test-user.keytab.
Jun 20 23:41:18 dist-test-slave-tcgx krb5kdc[22459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998878, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:41:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:41:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 23:41:18.732107 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781998525501163-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781998525501163-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781998525501163-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40619
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40619
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:41:19.076062 22472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:41:19.076459 22472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:41:19.076634 22472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:41:19.084179 22472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:41:19.084372 22472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:41:19.084479 22472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:41:19.084592 22472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:41:19.084668 22472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:41:19.095134 22472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781998525501163-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781998525501163-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40619
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781998525501163-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:41:19.096594 22472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:41:19.097965 22472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:41:19.110965 22477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:41:19.111024 22478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:41:19.110903 22480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:41:19.112313 22472 server_base.cc:1034] running on GCE node
I20260620 23:41:19.113270 22472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:41:19.115033 22472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:41:19.116299 22472 hybrid_clock.cc:648] HybridClock initialized: now 1781998879116269 us; error 75 us; skew 500 ppm
I20260620 23:41:19.116712 22472 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:41:19 dist-test-slave-tcgx krb5kdc[22459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998879, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:41:19.120138 22472 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 23:41:19.121958 22472 webserver.cc:460] Webserver started at http://127.0.97.62:46339/ using document root <none> and password file <none>
I20260620 23:41:19.122740 22472 fs_manager.cc:359] Metadata directory not provided
I20260620 23:41:19.122897 22472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:41:19.123282 22472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:41:19.126505 22472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781998525501163-388-0/minicluster-data/master-0/data/instance:
uuid: "796a722877ed4d66bc5350cc4b49baa5"
format_stamp: "Formatted at 2026-06-20 23:41:19 on dist-test-slave-tcgx"
server_key: "0c05c8c3091948d196a273093c9cccd0"
server_key_iv: "343c68675ea0d480b31ae89805895535"
server_key_version: "encryptionkey@0"
I20260620 23:41:19.127380 22472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781998525501163-388-0/minicluster-data/master-0/wal/instance:
uuid: "796a722877ed4d66bc5350cc4b49baa5"
format_stamp: "Formatted at 2026-06-20 23:41:19 on dist-test-slave-tcgx"
server_key: "0c05c8c3091948d196a273093c9cccd0"
server_key_iv: "343c68675ea0d480b31ae89805895535"
server_key_version: "encryptionkey@0"
I20260620 23:41:19.133145 22472 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 23:41:19.137212 22487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:41:19.138787 22472 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:41:19.139050 22472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781998525501163-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781998525501163-388-0/minicluster-data/master-0/wal
uuid: "796a722877ed4d66bc5350cc4b49baa5"
format_stamp: "Formatted at 2026-06-20 23:41:19 on dist-test-slave-tcgx"
server_key: "0c05c8c3091948d196a273093c9cccd0"
server_key_iv: "343c68675ea0d480b31ae89805895535"
server_key_version: "encryptionkey@0"
I20260620 23:41:19.139300 22472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781998525501163-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781998525501163-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781998525501163-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:41:19.161381 22472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:41:19.166373 22472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:41:19.166641 22472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:41:19.185947 22472 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40619
I20260620 23:41:19.185982 22538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40619 every 8 connection(s)
I20260620 23:41:19.187594 22472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:41:19.189141 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 22472
I20260620 23:41:19.189337 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781998525501163-388-0/minicluster-data/master-0/wal/instance
I20260620 23:41:19.189801 388 external_mini_cluster.cc:1428] Setting key 262fe2e9233362fbbc88592316b6e6fa
I20260620 23:41:19.192847 22539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 23:41:19 dist-test-slave-tcgx krb5kdc[22459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998878, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:41:19.205627 22539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 796a722877ed4d66bc5350cc4b49baa5: Bootstrap starting.
I20260620 23:41:19.209746 22539 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 796a722877ed4d66bc5350cc4b49baa5: Neither blocks nor log segments found. Creating new log.
I20260620 23:41:19.211112 22539 log.cc:824] T 00000000000000000000000000000000 P 796a722877ed4d66bc5350cc4b49baa5: Log is configured to *not* fsync() on all Append() calls
I20260620 23:41:19.213789 22542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:19.191814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48823 (local address 127.0.97.62:40619)
0620 23:41:19.192707 (+ 893us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:19.192719 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:19.192764 (+ 45us) server_negotiation.cc:407] Connection header received
0620 23:41:19.194476 (+ 1712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:19.194499 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:19.194927 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:19.195465 (+ 538us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:19.197086 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:19.198504 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:19.199953 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:19.200666 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:19.200880 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:19.205438 (+ 4558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:41:19.205447 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:41:19.205464 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:41:19.205509 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:41:19.209086 (+ 3577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:19.209905 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:19.209916 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:19.209926 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:19.210136 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:19.210560 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:19.210564 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:19.210566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:19.211214 (+ 648us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:41:19.211506 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0620 23:41:19.212142 (+ 636us) server_negotiation.cc:299] Negotiation successful
0620 23:41:19.212723 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":428,"threads_started":1}
I20260620 23:41:19.214861 22539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 796a722877ed4d66bc5350cc4b49baa5: No bootstrap required, opened a new log
I20260620 23:41:19.217602 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:41:19.223629 22539 raft_consensus.cc:348] T 00000000000000000000000000000000 P 796a722877ed4d66bc5350cc4b49baa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "796a722877ed4d66bc5350cc4b49baa5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40619 } }
I20260620 23:41:19.224072 22539 raft_consensus.cc:374] T 00000000000000000000000000000000 P 796a722877ed4d66bc5350cc4b49baa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:41:19.224184 22539 raft_consensus.cc:729] T 00000000000000000000000000000000 P 796a722877ed4d66bc5350cc4b49baa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 796a722877ed4d66bc5350cc4b49baa5, State: Initialized, Role: FOLLOWER
I20260620 23:41:19.224840 22539 consensus_queue.cc:260] T 00000000000000000000000000000000 P 796a722877ed4d66bc5350cc4b49baa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "796a722877ed4d66bc5350cc4b49baa5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40619 } }
I20260620 23:41:19.225111 22539 raft_consensus.cc:388] T 00000000000000000000000000000000 P 796a722877ed4d66bc5350cc4b49baa5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:41:19.225284 22539 raft_consensus.cc:482] T 00000000000000000000000000000000 P 796a722877ed4d66bc5350cc4b49baa5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:41:19.225493 22539 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 796a722877ed4d66bc5350cc4b49baa5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:41:19.227862 22539 raft_consensus.cc:504] T 00000000000000000000000000000000 P 796a722877ed4d66bc5350cc4b49baa5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "796a722877ed4d66bc5350cc4b49baa5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40619 } }
I20260620 23:41:19.228428 22539 leader_election.cc:302] T 00000000000000000000000000000000 P 796a722877ed4d66bc5350cc4b49baa5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 796a722877ed4d66bc5350cc4b49baa5; no voters:
I20260620 23:41:19.229228 22539 leader_election.cc:288] T 00000000000000000000000000000000 P 796a722877ed4d66bc5350cc4b49baa5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:41:19.229333 22549 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 796a722877ed4d66bc5350cc4b49baa5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:41:19.231241 22549 raft_consensus.cc:686] T 00000000000000000000000000000000 P 796a722877ed4d66bc5350cc4b49baa5 [term 1 LEADER]: Becoming Leader. State: Replica: 796a722877ed4d66bc5350cc4b49baa5, State: Running, Role: LEADER
I20260620 23:41:19.231983 22539 sys_catalog.cc:564] T 00000000000000000000000000000000 P 796a722877ed4d66bc5350cc4b49baa5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:41:19.231909 22549 consensus_queue.cc:237] T 00000000000000000000000000000000 P 796a722877ed4d66bc5350cc4b49baa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "796a722877ed4d66bc5350cc4b49baa5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40619 } }
I20260620 23:41:19.235131 22551 sys_catalog.cc:455] T 00000000000000000000000000000000 P 796a722877ed4d66bc5350cc4b49baa5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 796a722877ed4d66bc5350cc4b49baa5. Latest consensus state: current_term: 1 leader_uuid: "796a722877ed4d66bc5350cc4b49baa5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "796a722877ed4d66bc5350cc4b49baa5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40619 } } }
I20260620 23:41:19.235641 22551 sys_catalog.cc:458] T 00000000000000000000000000000000 P 796a722877ed4d66bc5350cc4b49baa5 [sys.catalog]: This master's current role is: LEADER
I20260620 23:41:19.237870 22550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 796a722877ed4d66bc5350cc4b49baa5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "796a722877ed4d66bc5350cc4b49baa5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "796a722877ed4d66bc5350cc4b49baa5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40619 } } }
I20260620 23:41:19.238301 22550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 796a722877ed4d66bc5350cc4b49baa5 [sys.catalog]: This master's current role is: LEADER
I20260620 23:41:19.240006 22542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:19.222232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53993 (local address 127.0.97.62:40619)
0620 23:41:19.222510 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:19.222520 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:19.223091 (+ 571us) server_negotiation.cc:407] Connection header received
0620 23:41:19.223305 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:19.223317 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:19.223383 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:19.223488 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:19.225066 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:19.226011 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:19.227362 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:19.227840 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:19.227991 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:19.230885 (+ 2894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:41:19.230899 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:41:19.230902 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:41:19.230992 (+ 90us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:41:19.234922 (+ 3930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:19.236099 (+ 1177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:19.236108 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:19.236114 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:19.236509 (+ 395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:19.237164 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:19.237172 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:19.237178 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:19.237680 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:41:19.239102 (+ 1422us) server_negotiation.cc:1035] Waiting for connection context
0620 23:41:19.239282 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 23:41:19.239551 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 23:41:19.240602 22556 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:41:19.246563 22556 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:41:19.257036 22556 catalog_manager.cc:1347] Generated new cluster ID: 6a8ce1966aa04fe7ac4ed9e09856bcb8
I20260620 23:41:19.257261 22556 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:41:19.298180 22556 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:41:19.299768 22556 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:41:19.310321 22556 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 796a722877ed4d66bc5350cc4b49baa5: Generated new TSK 0
I20260620 23:41:19.311506 22556 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:41:19.318425 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 22472
I20260620 23:41:19.319772 22534 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:41:19.415033 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 22472
2026-06-20T23:41:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2063 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781998525501163-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:41:19 dist-test-slave-tcgx krb5kdc[22571](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:41:19 dist-test-slave-tcgx krb5kdc[22571](info): set up 2 sockets
Jun 20 23:41:19 dist-test-slave-tcgx krb5kdc[22571](info): commencing operation
krb5kdc: starting...
W20260620 23:41:20.551412 388 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.003s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781998525501163-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781998525501163-388-0/krb5kdc/test-user.keytab.
Jun 20 23:41:20 dist-test-slave-tcgx krb5kdc[22571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:41:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:41:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 23:41:20.822268 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781998525501163-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781998525501163-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781998525501163-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33489
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33489
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:41:21.160317 22584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:41:21.160645 22584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:41:21.160846 22584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:41:21.168197 22584 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:41:21.168339 22584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:41:21.168432 22584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:41:21.168550 22584 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:41:21.168630 22584 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:41:21.178673 22584 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781998525501163-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781998525501163-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33489
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781998525501163-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:41:21.180116 22584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:41:21.181489 22584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:41:21.193720 22590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:41:21.193707 22589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:41:21.193768 22592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:41:21.194581 22584 server_base.cc:1034] running on GCE node
I20260620 23:41:21.195477 22584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:41:21.196892 22584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:41:21.198096 22584 hybrid_clock.cc:648] HybridClock initialized: now 1781998881198038 us; error 73 us; skew 500 ppm
I20260620 23:41:21.198472 22584 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:41:21 dist-test-slave-tcgx krb5kdc[22571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:41:21.201726 22584 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 23:41:21.203375 22584 webserver.cc:460] Webserver started at http://127.0.97.62:37111/ using document root <none> and password file <none>
I20260620 23:41:21.204097 22584 fs_manager.cc:359] Metadata directory not provided
I20260620 23:41:21.204245 22584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:41:21.204566 22584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:41:21.207712 22584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781998525501163-388-0/minicluster-data/master-0/data/instance:
uuid: "36b9afc62df1407289ca2ba39f4e05f4"
format_stamp: "Formatted at 2026-06-20 23:41:21 on dist-test-slave-tcgx"
server_key: "206e801d86659174e5a239f75f53bf6b"
server_key_iv: "9d64a0ec3b316a72f4b6b3d6d43b587d"
server_key_version: "encryptionkey@0"
I20260620 23:41:21.208498 22584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781998525501163-388-0/minicluster-data/master-0/wal/instance:
uuid: "36b9afc62df1407289ca2ba39f4e05f4"
format_stamp: "Formatted at 2026-06-20 23:41:21 on dist-test-slave-tcgx"
server_key: "206e801d86659174e5a239f75f53bf6b"
server_key_iv: "9d64a0ec3b316a72f4b6b3d6d43b587d"
server_key_version: "encryptionkey@0"
I20260620 23:41:21.213784 22584 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 23:41:21.217576 22599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:41:21.218923 22584 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:41:21.219201 22584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781998525501163-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781998525501163-388-0/minicluster-data/master-0/wal
uuid: "36b9afc62df1407289ca2ba39f4e05f4"
format_stamp: "Formatted at 2026-06-20 23:41:21 on dist-test-slave-tcgx"
server_key: "206e801d86659174e5a239f75f53bf6b"
server_key_iv: "9d64a0ec3b316a72f4b6b3d6d43b587d"
server_key_version: "encryptionkey@0"
I20260620 23:41:21.219446 22584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781998525501163-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781998525501163-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781998525501163-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:41:21.232169 22584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:41:21.236557 22584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:41:21.236861 22584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:41:21.256171 22584 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33489
I20260620 23:41:21.256179 22650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33489 every 8 connection(s)
I20260620 23:41:21.257627 22584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:41:21.258028 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 22584
I20260620 23:41:21.258199 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781998525501163-388-0/minicluster-data/master-0/wal/instance
I20260620 23:41:21.258764 388 external_mini_cluster.cc:1428] Setting key 0a44aa37ac4fbb5ecf8813dd75799541
I20260620 23:41:21.262487 22651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:41:21.274633 22654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:21.260719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58589 (local address 127.0.97.62:33489)
0620 23:41:21.261505 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:21.261520 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:21.261572 (+ 52us) server_negotiation.cc:407] Connection header received
0620 23:41:21.263944 (+ 2372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:21.263973 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:21.264501 (+ 528us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:21.264975 (+ 474us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:21.265517 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:21.267570 (+ 2053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:21.269053 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:21.270084 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:21.270259 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:21.271614 (+ 1355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:41:21.271624 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:41:21.271644 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:41:21.272441 (+ 797us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:41:21.272531 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:41:21.272567 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:41:21.272687 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 23:41:21.272927 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 23:41:21.273225 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":318,"threads_started":1}
I20260620 23:41:21.276746 22651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36b9afc62df1407289ca2ba39f4e05f4: Bootstrap starting.
I20260620 23:41:21.278671 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:41:21.280268 22651 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36b9afc62df1407289ca2ba39f4e05f4: Neither blocks nor log segments found. Creating new log.
I20260620 23:41:21.281250 22651 log.cc:824] T 00000000000000000000000000000000 P 36b9afc62df1407289ca2ba39f4e05f4: Log is configured to *not* fsync() on all Append() calls
I20260620 23:41:21.284394 22651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36b9afc62df1407289ca2ba39f4e05f4: No bootstrap required, opened a new log
I20260620 23:41:21.289812 22654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:21.283013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60065 (local address 127.0.97.62:33489)
0620 23:41:21.283209 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:21.283218 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:21.283773 (+ 555us) server_negotiation.cc:407] Connection header received
0620 23:41:21.283969 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:21.283973 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:21.284030 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:21.284108 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:21.284741 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:21.285478 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:21.287357 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:21.287817 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:21.287915 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:21.289081 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:41:21.289085 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:41:21.289088 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:41:21.289104 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:41:21.289140 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:41:21.289154 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:41:21.289224 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 23:41:21.289387 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 23:41:21.289461 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 23:41:21.293704 22651 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36b9afc62df1407289ca2ba39f4e05f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36b9afc62df1407289ca2ba39f4e05f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33489 } }
I20260620 23:41:21.294128 22651 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36b9afc62df1407289ca2ba39f4e05f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:41:21.294255 22651 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36b9afc62df1407289ca2ba39f4e05f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36b9afc62df1407289ca2ba39f4e05f4, State: Initialized, Role: FOLLOWER
I20260620 23:41:21.294919 22651 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36b9afc62df1407289ca2ba39f4e05f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36b9afc62df1407289ca2ba39f4e05f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33489 } }
I20260620 23:41:21.295228 22651 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36b9afc62df1407289ca2ba39f4e05f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:41:21.295367 22651 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36b9afc62df1407289ca2ba39f4e05f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:41:21.295531 22651 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36b9afc62df1407289ca2ba39f4e05f4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:41:21.297976 22651 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36b9afc62df1407289ca2ba39f4e05f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36b9afc62df1407289ca2ba39f4e05f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33489 } }
I20260620 23:41:21.298516 22651 leader_election.cc:302] T 00000000000000000000000000000000 P 36b9afc62df1407289ca2ba39f4e05f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36b9afc62df1407289ca2ba39f4e05f4; no voters:
I20260620 23:41:21.299203 22651 leader_election.cc:288] T 00000000000000000000000000000000 P 36b9afc62df1407289ca2ba39f4e05f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:41:21.299364 22661 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36b9afc62df1407289ca2ba39f4e05f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:41:21.300424 22661 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36b9afc62df1407289ca2ba39f4e05f4 [term 1 LEADER]: Becoming Leader. State: Replica: 36b9afc62df1407289ca2ba39f4e05f4, State: Running, Role: LEADER
I20260620 23:41:21.301069 22661 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36b9afc62df1407289ca2ba39f4e05f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36b9afc62df1407289ca2ba39f4e05f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33489 } }
I20260620 23:41:21.301618 22651 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36b9afc62df1407289ca2ba39f4e05f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:41:21.304091 22662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36b9afc62df1407289ca2ba39f4e05f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36b9afc62df1407289ca2ba39f4e05f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36b9afc62df1407289ca2ba39f4e05f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33489 } } }
I20260620 23:41:21.304502 22662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36b9afc62df1407289ca2ba39f4e05f4 [sys.catalog]: This master's current role is: LEADER
I20260620 23:41:21.304205 22663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36b9afc62df1407289ca2ba39f4e05f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36b9afc62df1407289ca2ba39f4e05f4. Latest consensus state: current_term: 1 leader_uuid: "36b9afc62df1407289ca2ba39f4e05f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36b9afc62df1407289ca2ba39f4e05f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33489 } } }
I20260620 23:41:21.304664 22663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36b9afc62df1407289ca2ba39f4e05f4 [sys.catalog]: This master's current role is: LEADER
I20260620 23:41:21.314950 22667 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:41:21.320623 22667 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:41:21.330044 22667 catalog_manager.cc:1347] Generated new cluster ID: f8f5c582a3604c58b619bf3e0b88dcda
I20260620 23:41:21.330178 22667 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:41:21.348412 22667 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:41:21.349771 22667 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:41:21.357997 22667 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36b9afc62df1407289ca2ba39f4e05f4: Generated new TSK 0
I20260620 23:41:21.359207 22667 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:41:21.376550 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 22584
W20260620 23:41:21.377699 22646 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:58589
W20260620 23:41:21.378444 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 22584: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:41:21.378585 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 22584
2026-06-20T23:41:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1962 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781998525501163-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:41:21 dist-test-slave-tcgx krb5kdc[22682](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:41:21 dist-test-slave-tcgx krb5kdc[22682](info): set up 2 sockets
Jun 20 23:41:21 dist-test-slave-tcgx krb5kdc[22682](info): commencing operation
krb5kdc: starting...
W20260620 23:41:22.507680 388 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781998525501163-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781998525501163-388-0/krb5kdc/test-user.keytab.
Jun 20 23:41:22 dist-test-slave-tcgx krb5kdc[22682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:41:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:41:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 23:41:22.779083 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781998525501163-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781998525501163-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781998525501163-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43279
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:41:23.122589 22695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:41:23.122928 22695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:41:23.123135 22695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:41:23.130620 22695 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:41:23.130752 22695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:41:23.130839 22695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:41:23.130961 22695 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:41:23.131052 22695 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:41:23.141288 22695 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781998525501163-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781998525501163-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781998525501163-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:41:23.142724 22695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:41:23.144158 22695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:41:23.156347 22701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:41:23.156436 22700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:41:23.156411 22703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:41:23.157778 22695 server_base.cc:1034] running on GCE node
I20260620 23:41:23.158669 22695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:41:23.160243 22695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:41:23.161487 22695 hybrid_clock.cc:648] HybridClock initialized: now 1781998883161381 us; error 62 us; skew 500 ppm
I20260620 23:41:23.161864 22695 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:41:23 dist-test-slave-tcgx krb5kdc[22682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998883, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:41:23.165149 22695 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 23:41:23.166639 22695 webserver.cc:460] Webserver started at http://127.0.97.62:43017/ using document root <none> and password file <none>
I20260620 23:41:23.167408 22695 fs_manager.cc:359] Metadata directory not provided
I20260620 23:41:23.167522 22695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:41:23.167783 22695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:41:23.170804 22695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781998525501163-388-0/minicluster-data/master-0/data/instance:
uuid: "2531605eb78d4c329adebe29c78a7ba5"
format_stamp: "Formatted at 2026-06-20 23:41:23 on dist-test-slave-tcgx"
server_key: "ad703ac7d0f8f180b74d6f7a8f11cfc8"
server_key_iv: "f256c9dc2bd062a6383de19a3fa2d173"
server_key_version: "encryptionkey@0"
I20260620 23:41:23.171643 22695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781998525501163-388-0/minicluster-data/master-0/wal/instance:
uuid: "2531605eb78d4c329adebe29c78a7ba5"
format_stamp: "Formatted at 2026-06-20 23:41:23 on dist-test-slave-tcgx"
server_key: "ad703ac7d0f8f180b74d6f7a8f11cfc8"
server_key_iv: "f256c9dc2bd062a6383de19a3fa2d173"
server_key_version: "encryptionkey@0"
I20260620 23:41:23.177129 22695 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:41:23.181038 22710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:41:23.182662 22695 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:41:23.182906 22695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781998525501163-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781998525501163-388-0/minicluster-data/master-0/wal
uuid: "2531605eb78d4c329adebe29c78a7ba5"
format_stamp: "Formatted at 2026-06-20 23:41:23 on dist-test-slave-tcgx"
server_key: "ad703ac7d0f8f180b74d6f7a8f11cfc8"
server_key_iv: "f256c9dc2bd062a6383de19a3fa2d173"
server_key_version: "encryptionkey@0"
I20260620 23:41:23.183213 22695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781998525501163-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781998525501163-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781998525501163-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:41:23.189913 22695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:41:23.194670 22695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:41:23.195070 22695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:41:23.215917 22695 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43279
I20260620 23:41:23.215965 22761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43279 every 8 connection(s)
I20260620 23:41:23.217528 22695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:41:23.221712 22762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:41:23.225636 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 22695
I20260620 23:41:23.225818 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781998525501163-388-0/minicluster-data/master-0/wal/instance
I20260620 23:41:23.226370 388 external_mini_cluster.cc:1428] Setting key 875a10edfad2dbaa9d674550a53be5e2
I20260620 23:41:23.235246 22765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:23.228444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33977 (local address 127.0.97.62:43279)
0620 23:41:23.229358 (+ 914us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:23.229371 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:23.229415 (+ 44us) server_negotiation.cc:407] Connection header received
0620 23:41:23.231341 (+ 1926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:23.231368 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:23.231794 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:23.232197 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:23.233381 (+ 1184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:41:23.233389 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:41:23.233403 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:41:23.233427 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:41:23.233497 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:41:23.233522 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:41:23.233708 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 23:41:23.233835 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 23:41:23.234090 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":676,"thread_start_us":416,"threads_started":1}
I20260620 23:41:23.235582 22762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2531605eb78d4c329adebe29c78a7ba5: Bootstrap starting.
I20260620 23:41:23.239133 22762 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2531605eb78d4c329adebe29c78a7ba5: Neither blocks nor log segments found. Creating new log.
I20260620 23:41:23.239727 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:41:23.240643 22762 log.cc:824] T 00000000000000000000000000000000 P 2531605eb78d4c329adebe29c78a7ba5: Log is configured to *not* fsync() on all Append() calls
I20260620 23:41:23.244444 22762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2531605eb78d4c329adebe29c78a7ba5: No bootstrap required, opened a new log
I20260620 23:41:23.247874 22765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:23.245016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36059 (local address 127.0.97.62:43279)
0620 23:41:23.245224 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:23.245232 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:23.245748 (+ 516us) server_negotiation.cc:407] Connection header received
0620 23:41:23.245900 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:23.245905 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:23.245970 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:23.246095 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:23.247145 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:41:23.247151 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:41:23.247158 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:41:23.247182 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:41:23.247235 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:41:23.247257 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:41:23.247388 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 23:41:23.247475 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 23:41:23.247565 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 23:41:23.253044 22762 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2531605eb78d4c329adebe29c78a7ba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2531605eb78d4c329adebe29c78a7ba5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43279 } }
I20260620 23:41:23.253505 22762 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2531605eb78d4c329adebe29c78a7ba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:41:23.253613 22762 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2531605eb78d4c329adebe29c78a7ba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2531605eb78d4c329adebe29c78a7ba5, State: Initialized, Role: FOLLOWER
I20260620 23:41:23.254338 22762 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2531605eb78d4c329adebe29c78a7ba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2531605eb78d4c329adebe29c78a7ba5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43279 } }
I20260620 23:41:23.254590 22762 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2531605eb78d4c329adebe29c78a7ba5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:41:23.254729 22762 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2531605eb78d4c329adebe29c78a7ba5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:41:23.254935 22762 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2531605eb78d4c329adebe29c78a7ba5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:41:23.257387 22762 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2531605eb78d4c329adebe29c78a7ba5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2531605eb78d4c329adebe29c78a7ba5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43279 } }
I20260620 23:41:23.257992 22762 leader_election.cc:302] T 00000000000000000000000000000000 P 2531605eb78d4c329adebe29c78a7ba5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2531605eb78d4c329adebe29c78a7ba5; no voters:
I20260620 23:41:23.258700 22762 leader_election.cc:288] T 00000000000000000000000000000000 P 2531605eb78d4c329adebe29c78a7ba5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:41:23.258824 22772 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2531605eb78d4c329adebe29c78a7ba5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:41:23.259588 22772 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2531605eb78d4c329adebe29c78a7ba5 [term 1 LEADER]: Becoming Leader. State: Replica: 2531605eb78d4c329adebe29c78a7ba5, State: Running, Role: LEADER
I20260620 23:41:23.260114 22772 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2531605eb78d4c329adebe29c78a7ba5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2531605eb78d4c329adebe29c78a7ba5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43279 } }
I20260620 23:41:23.260946 22762 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2531605eb78d4c329adebe29c78a7ba5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:41:23.263213 22774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2531605eb78d4c329adebe29c78a7ba5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2531605eb78d4c329adebe29c78a7ba5. Latest consensus state: current_term: 1 leader_uuid: "2531605eb78d4c329adebe29c78a7ba5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2531605eb78d4c329adebe29c78a7ba5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43279 } } }
I20260620 23:41:23.263378 22773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2531605eb78d4c329adebe29c78a7ba5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2531605eb78d4c329adebe29c78a7ba5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2531605eb78d4c329adebe29c78a7ba5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43279 } } }
I20260620 23:41:23.263796 22774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2531605eb78d4c329adebe29c78a7ba5 [sys.catalog]: This master's current role is: LEADER
I20260620 23:41:23.263867 22773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2531605eb78d4c329adebe29c78a7ba5 [sys.catalog]: This master's current role is: LEADER
I20260620 23:41:23.267151 22779 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:41:23.272820 22779 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:41:23.282645 22779 catalog_manager.cc:1347] Generated new cluster ID: f3745dbd0f8547b7b967bce4b50ea9f3
I20260620 23:41:23.282791 22779 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:41:23.303169 22779 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:41:23.303650 22779 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:41:23.315590 22779 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2531605eb78d4c329adebe29c78a7ba5: Generated new TSK 0
I20260620 23:41:23.316570 22779 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:41:23.328948 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 22695
W20260620 23:41:23.330118 22757 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33977
W20260620 23:41:23.330907 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 22695: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:41:23.331125 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 22695
2026-06-20T23:41:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1952 ms)
[----------] 18 tests from AuthTokenIssuingTest (35846 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:41:23 dist-test-slave-tcgx krb5kdc[22793](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:41:23 dist-test-slave-tcgx krb5kdc[22793](info): set up 2 sockets
Jun 20 23:41:23 dist-test-slave-tcgx krb5kdc[22793](info): commencing operation
krb5kdc: starting...
W20260620 23:41:24.455592 388 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/krb5kdc/test-user.keytab.
Jun 20 23:41:24 dist-test-slave-tcgx krb5kdc[22793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:41:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:41:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 23:41:24.726393 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46097
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46097,127.0.97.61:34587,127.0.97.60:46709
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:41:25.075140 22806 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 23:41:25.075575 22806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:41:25.075708 22806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:41:25.075815 22806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:41:25.083580 22806 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:41:25.083691 22806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:41:25.083781 22806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:41:25.083882 22806 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:41:25.083993 22806 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:41:25.095052 22806 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46097,127.0.97.61:34587,127.0.97.60:46709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:41:25.096539 22806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:41:25.097944 22806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:41:25.111258 22811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:41:25.111467 22812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:41:25.111511 22814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:41:25.112478 22806 server_base.cc:1034] running on GCE node
I20260620 23:41:25.113451 22806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:41:25.115226 22806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:41:25.116484 22806 hybrid_clock.cc:648] HybridClock initialized: now 1781998885116430 us; error 41 us; skew 500 ppm
I20260620 23:41:25.116907 22806 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:41:25 dist-test-slave-tcgx krb5kdc[22793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:41:25.120385 22806 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 23:41:25.122071 22806 webserver.cc:460] Webserver started at http://127.0.97.62:35001/ using document root <none> and password file <none>
I20260620 23:41:25.122874 22806 fs_manager.cc:359] Metadata directory not provided
I20260620 23:41:25.123044 22806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:41:25.123394 22806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:41:25.126714 22806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/minicluster-data/master-0/data/instance:
uuid: "ad1d7e3982a542e4a95b44470e9af1aa"
format_stamp: "Formatted at 2026-06-20 23:41:25 on dist-test-slave-tcgx"
server_key: "8a12b99744dc30048a21a5eabacb234b"
server_key_iv: "13f9eecc06acd93a19836b29f9edd446"
server_key_version: "encryptionkey@0"
I20260620 23:41:25.127674 22806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/minicluster-data/master-0/wal/instance:
uuid: "ad1d7e3982a542e4a95b44470e9af1aa"
format_stamp: "Formatted at 2026-06-20 23:41:25 on dist-test-slave-tcgx"
server_key: "8a12b99744dc30048a21a5eabacb234b"
server_key_iv: "13f9eecc06acd93a19836b29f9edd446"
server_key_version: "encryptionkey@0"
I20260620 23:41:25.133507 22806 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 23:41:25.137624 22821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:41:25.139441 22806 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 23:41:25.139670 22806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/minicluster-data/master-0/wal
uuid: "ad1d7e3982a542e4a95b44470e9af1aa"
format_stamp: "Formatted at 2026-06-20 23:41:25 on dist-test-slave-tcgx"
server_key: "8a12b99744dc30048a21a5eabacb234b"
server_key_iv: "13f9eecc06acd93a19836b29f9edd446"
server_key_version: "encryptionkey@0"
I20260620 23:41:25.139953 22806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:41:25.163717 22806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:41:25.168558 22806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:41:25.168865 22806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:41:25.188488 22806 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46097
I20260620 23:41:25.188534 22872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46097 every 8 connection(s)
I20260620 23:41:25.189973 22806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:41:25.194013 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 22806
I20260620 23:41:25.194293 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/minicluster-data/master-0/wal/instance
I20260620 23:41:25.194340 22873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:41:25.195024 388 external_mini_cluster.cc:1428] Setting key a03893bd6ef61a2ea00b8fc090e10961
I20260620 23:41:25.201857 22873 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46097 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
Jun 20 23:41:25 dist-test-slave-tcgx krb5kdc[22793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:41:25.232219 22877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:25.205871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46097 (local address 127.0.0.1:40394)
0620 23:41:25.207390 (+ 1519us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:25.207448 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 23:41:25.209601 (+ 2153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:41:25.210706 (+ 1105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:41:25.210712 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:41:25.211454 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:41:25.211826 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:41:25.211845 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:41:25.214358 (+ 2513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:41:25.214363 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:41:25.215701 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:41:25.215715 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:41:25.217207 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:41:25.217213 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:41:25.217356 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:25.218097 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:41:25.218130 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:41:25.223090 (+ 4960us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:41:25.227614 (+ 4524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:41:25.227625 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:41:25.227655 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:41:25.228082 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:41:25.228588 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:41:25.228595 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:41:25.228598 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:41:25.228769 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:41:25.229609 (+ 840us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:41:25.229619 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:41:25.229838 (+ 219us) client_negotiation.cc:769] Sending connection context
0620 23:41:25.230102 (+ 264us) client_negotiation.cc:240] Negotiation successful
0620 23:41:25.230788 (+ 686us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1218,"mutex_wait_us":218,"thread_start_us":982,"threads_started":1}
I20260620 23:41:25.232250 22876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:25.205871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40394 (local address 127.0.97.62:46097)
0620 23:41:25.206877 (+ 1006us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:25.206899 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:25.207609 (+ 710us) server_negotiation.cc:407] Connection header received
0620 23:41:25.210060 (+ 2451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:25.210072 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:25.210431 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:25.210674 (+ 243us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:25.212080 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:25.214156 (+ 2076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:25.215869 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:25.216955 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:25.217191 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:25.223404 (+ 6213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:41:25.223419 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:41:25.223443 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:41:25.223498 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:41:25.227430 (+ 3932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:25.228257 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:25.228263 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:25.228266 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:25.228439 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:25.228891 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:25.228897 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:25.228905 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:25.229454 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:41:25.229684 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0620 23:41:25.230242 (+ 558us) server_negotiation.cc:299] Negotiation successful
0620 23:41:25.230670 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":723,"thread_start_us":466,"threads_started":1}
I20260620 23:41:25.235628 22873 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34587 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:41:25.236829 22877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:25.236380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34587 (local address 127.0.0.1:45822)
0620 23:41:25.236594 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:25.236713 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260620 23:41:25.237129 22822 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34587: connect: Connection refused (error 111)
I20260620 23:41:25.238765 22877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:25.238385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34587 (local address 127.0.0.1:45834)
0620 23:41:25.238552 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:25.238632 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 23:41:25.239029 22873 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:34587: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34587: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260620 23:41:25.256572 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:34587
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46097,127.0.97.61:34587,127.0.97.60:46709
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:41:25.288417 22873 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34587 } attempt: 1
I20260620 23:41:25.289788 22877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:25.289232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34587 (local address 127.0.0.1:45850)
0620 23:41:25.289475 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:25.289590 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260620 23:41:25.291055 22877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:25.290593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34587 (local address 127.0.0.1:45864)
0620 23:41:25.290766 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:25.290855 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260620 23:41:25.291283 22873 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:34587: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34587: connect: Connection refused (error 111)
I20260620 23:41:25.359536 22873 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34587 } attempt: 2
I20260620 23:41:25.360884 22877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:25.360308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34587 (local address 127.0.0.1:45866)
0620 23:41:25.360592 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:25.360707 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260620 23:41:25.361989 22877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:25.361572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34587 (local address 127.0.0.1:45872)
0620 23:41:25.361728 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:25.361827 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 23:41:25.362196 22873 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:34587: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34587: connect: Connection refused (error 111)
I20260620 23:41:25.453466 22873 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34587 } attempt: 3
I20260620 23:41:25.454782 22877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:25.454237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34587 (local address 127.0.0.1:50164)
0620 23:41:25.454478 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:25.454597 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260620 23:41:25.455996 22877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:25.455548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34587 (local address 127.0.0.1:50170)
0620 23:41:25.455715 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:25.455804 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260620 23:41:25.456229 22873 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:34587: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34587: connect: Connection refused (error 111)
I20260620 23:41:25.599503 22873 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34587 } attempt: 4
I20260620 23:41:25.600816 22877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:25.600196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34587 (local address 127.0.0.1:50180)
0620 23:41:25.600543 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:25.600655 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183}
I20260620 23:41:25.602056 22877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:25.601617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34587 (local address 127.0.0.1:50184)
0620 23:41:25.601782 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:25.601899 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 23:41:25.602381 22873 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:34587: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34587: connect: Connection refused (error 111)
W20260620 23:41:25.609200 22880 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 23:41:25.609583 22880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:41:25.609687 22880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:41:25.609817 22880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:41:25.617326 22880 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:41:25.617437 22880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:41:25.617532 22880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:41:25.617622 22880 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:41:25.617735 22880 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:41:25.627913 22880 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46097,127.0.97.61:34587,127.0.97.60:46709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:34587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:41:25.629336 22880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:41:25.630793 22880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:41:25.643277 22888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:41:25.643822 22886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:41:25.643947 22885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:41:25.645208 22880 server_base.cc:1034] running on GCE node
I20260620 23:41:25.646381 22880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:41:25.648525 22880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:41:25.649834 22880 hybrid_clock.cc:648] HybridClock initialized: now 1781998885649734 us; error 47 us; skew 500 ppm
I20260620 23:41:25.650326 22880 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:41:25 dist-test-slave-tcgx krb5kdc[22793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:41:25.655108 22880 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260620 23:41:25.657269 22880 webserver.cc:460] Webserver started at http://127.0.97.61:46795/ using document root <none> and password file <none>
I20260620 23:41:25.658344 22880 fs_manager.cc:359] Metadata directory not provided
I20260620 23:41:25.658489 22880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:41:25.658908 22880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:41:25.663700 22880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/minicluster-data/master-1/data/instance:
uuid: "f2a4ad556e23425ebf8c2f9d0605a501"
format_stamp: "Formatted at 2026-06-20 23:41:25 on dist-test-slave-tcgx"
server_key: "7ff67633976b13bcb4d65002e8ff675e"
server_key_iv: "74ad3c5fd95d2ec16a11892436c0c8da"
server_key_version: "encryptionkey@0"
I20260620 23:41:25.665004 22880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/minicluster-data/master-1/wal/instance:
uuid: "f2a4ad556e23425ebf8c2f9d0605a501"
format_stamp: "Formatted at 2026-06-20 23:41:25 on dist-test-slave-tcgx"
server_key: "7ff67633976b13bcb4d65002e8ff675e"
server_key_iv: "74ad3c5fd95d2ec16a11892436c0c8da"
server_key_version: "encryptionkey@0"
I20260620 23:41:25.673401 22880 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.008s sys 0.000s
I20260620 23:41:25.679188 22895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:41:25.681278 22880 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 23:41:25.681499 22880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/minicluster-data/master-1/wal
uuid: "f2a4ad556e23425ebf8c2f9d0605a501"
format_stamp: "Formatted at 2026-06-20 23:41:25 on dist-test-slave-tcgx"
server_key: "7ff67633976b13bcb4d65002e8ff675e"
server_key_iv: "74ad3c5fd95d2ec16a11892436c0c8da"
server_key_version: "encryptionkey@0"
I20260620 23:41:25.681800 22880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:41:25.731576 22880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:41:25.736440 22880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:41:25.736706 22880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:41:25.756325 22880 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:34587
I20260620 23:41:25.756346 22946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:34587 every 8 connection(s)
I20260620 23:41:25.757838 22880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/minicluster-data/master-1/data/info.pb
I20260620 23:41:25.761758 22947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:41:25.764782 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 22880
I20260620 23:41:25.764972 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/minicluster-data/master-1/wal/instance
I20260620 23:41:25.765502 388 external_mini_cluster.cc:1428] Setting key 55dc5c19bd4139969efc7a28c2d54d74
I20260620 23:41:25.771191 22947 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46097 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:41:25 dist-test-slave-tcgx krb5kdc[22793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260620 23:41:25.797366 22949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:25.774875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55686 (local address 127.0.97.62:46097)
0620 23:41:25.775464 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:25.775473 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:25.776670 (+ 1197us) server_negotiation.cc:407] Connection header received
0620 23:41:25.779832 (+ 3162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:25.779836 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:25.779914 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:25.780006 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:25.782322 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:25.783118 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:25.785616 (+ 2498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:25.786013 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:25.786119 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:25.790705 (+ 4586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:41:25.790714 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:41:25.790720 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:41:25.790785 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:41:25.793801 (+ 3016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:25.794719 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:25.794730 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:25.794733 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:25.794858 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:25.795368 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:25.795377 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:25.795380 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:25.795716 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:41:25.795898 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 23:41:25.796698 (+ 800us) server_negotiation.cc:299] Negotiation successful
0620 23:41:25.796939 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":273,"threads_started":1}
I20260620 23:41:25.798362 22950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:25.775569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46097 (local address 127.0.0.1:55686)
0620 23:41:25.776454 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:25.776522 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0620 23:41:25.779441 (+ 2919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:41:25.780322 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:41:25.780337 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:41:25.781206 (+ 869us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:41:25.782081 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:41:25.782111 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:41:25.783284 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:41:25.783292 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:41:25.785442 (+ 2150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:41:25.785463 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:41:25.786276 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:41:25.786285 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:41:25.786661 (+ 376us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:25.787526 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:41:25.787564 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:41:25.790411 (+ 2847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:41:25.793984 (+ 3573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:41:25.793995 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:41:25.794014 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:41:25.794582 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:41:25.795022 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:41:25.795027 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:41:25.795031 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:41:25.795245 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:41:25.795862 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:41:25.795870 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:41:25.796282 (+ 412us) client_negotiation.cc:769] Sending connection context
0620 23:41:25.796592 (+ 310us) client_negotiation.cc:240] Negotiation successful
0620 23:41:25.797174 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":413,"threads_started":1}
I20260620 23:41:25.800305 22947 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34587 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:41:25 dist-test-slave-tcgx krb5kdc[22793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 23:41:25.815610 22950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:25.801189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34587 (local address 127.0.0.1:50186)
0620 23:41:25.801398 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:25.801421 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 23:41:25.801518 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:41:25.802536 (+ 1018us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:41:25.802541 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:41:25.803055 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:41:25.803193 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:41:25.803202 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:41:25.804640 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:41:25.804645 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:41:25.805490 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:41:25.805501 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:41:25.806347 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:41:25.806351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:41:25.806585 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:25.807269 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:41:25.807287 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:41:25.809709 (+ 2422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:41:25.813031 (+ 3322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:41:25.813037 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:41:25.813041 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:41:25.813398 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:41:25.813780 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:41:25.813785 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:41:25.813789 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:41:25.813927 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:41:25.814559 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:41:25.814563 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:41:25.814711 (+ 148us) client_negotiation.cc:769] Sending connection context
0620 23:41:25.814851 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 23:41:25.815152 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":45}
I20260620 23:41:25.815649 22952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:25.801488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50186 (local address 127.0.97.61:34587)
0620 23:41:25.802122 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:25.802129 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:25.802153 (+ 24us) server_negotiation.cc:407] Connection header received
0620 23:41:25.802196 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:25.802202 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:25.802387 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:25.802568 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:25.803369 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:25.804474 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:25.805644 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:25.806199 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:25.806335 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:25.809924 (+ 3589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:41:25.809937 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:41:25.809951 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:41:25.809980 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:41:25.812875 (+ 2895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:25.813531 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:25.813540 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:25.813544 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:25.813666 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:25.814030 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:25.814037 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:25.814040 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:25.814405 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:41:25.814629 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 23:41:25.815033 (+ 404us) server_negotiation.cc:299] Negotiation successful
0620 23:41:25.815258 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":481,"thread_start_us":341,"threads_started":1}
I20260620 23:41:25.818531 22947 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46709 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:41:25.819788 22950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:25.819302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46709 (local address 127.0.0.1:56704)
0620 23:41:25.819491 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:25.819629 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260620 23:41:25.820188 22896 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46709: connect: Connection refused (error 111)
I20260620 23:41:25.821811 22950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:25.821497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46709 (local address 127.0.0.1:56716)
0620 23:41:25.821630 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:25.821705 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 23:41:25.822067 22947 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46709: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46709: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260620 23:41:25.830219 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:46709
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46097,127.0.97.61:34587,127.0.97.60:46709
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:41:25.871393 22947 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46709 } attempt: 1
I20260620 23:41:25.872637 22950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:25.872177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46709 (local address 127.0.0.1:56718)
0620 23:41:25.872368 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:25.872468 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260620 23:41:25.873601 22950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:25.873254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46709 (local address 127.0.0.1:56720)
0620 23:41:25.873418 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:25.873490 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 23:41:25.873799 22947 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46709: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46709: connect: Connection refused (error 111)
I20260620 23:41:25.901646 22873 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34587 } attempt: 5
Jun 20 23:41:25 dist-test-slave-tcgx krb5kdc[22793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 23:41:25.916821 22877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:25.902405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34587 (local address 127.0.0.1:50200)
0620 23:41:25.902602 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:25.902625 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 23:41:25.902748 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:41:25.903245 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:41:25.903250 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:41:25.903638 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:41:25.903818 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:41:25.903826 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:41:25.905273 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:41:25.905277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:41:25.906190 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:41:25.906202 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:41:25.907242 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:41:25.907246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:41:25.907409 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:25.908072 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:41:25.908091 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:41:25.910254 (+ 2163us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:41:25.914207 (+ 3953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:41:25.914211 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:41:25.914214 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:41:25.914608 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:41:25.915150 (+ 542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:41:25.915153 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:41:25.915156 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:41:25.915287 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:41:25.916043 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:41:25.916052 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:41:25.916182 (+ 130us) client_negotiation.cc:769] Sending connection context
0620 23:41:25.916326 (+ 144us) client_negotiation.cc:240] Negotiation successful
0620 23:41:25.916565 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 23:41:25.917042 22952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:25.902529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50200 (local address 127.0.97.61:34587)
0620 23:41:25.902743 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:25.902752 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:25.902783 (+ 31us) server_negotiation.cc:407] Connection header received
0620 23:41:25.902878 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:25.902886 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:25.903039 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:25.903185 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:25.903981 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:25.905118 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:25.906347 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:25.907096 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:25.907243 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:25.910474 (+ 3231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:41:25.910481 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:41:25.910486 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:41:25.910530 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:41:25.913936 (+ 3406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:25.914757 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:25.914765 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:25.914772 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:25.915006 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:25.915409 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:25.915417 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:25.915421 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:25.915901 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:41:25.916097 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0620 23:41:25.916480 (+ 383us) server_negotiation.cc:299] Negotiation successful
0620 23:41:25.916720 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 23:41:25.918068 22873 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46709 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:41:25.919119 22877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:25.918670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46709 (local address 127.0.0.1:56728)
0620 23:41:25.918834 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:25.918916 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 23:41:25.920055 22877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:25.919722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46709 (local address 127.0.0.1:56744)
0620 23:41:25.919864 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:25.919954 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 23:41:25.920331 22873 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46709: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46709: connect: Connection refused (error 111)
I20260620 23:41:25.942009 22947 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46709 } attempt: 2
I20260620 23:41:25.943364 22950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:25.942828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46709 (local address 127.0.0.1:56760)
0620 23:41:25.943079 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:25.943176 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 23:41:25.944461 22950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:25.944070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46709 (local address 127.0.0.1:56764)
0620 23:41:25.944202 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:25.944306 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 23:41:25.944746 22947 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46709: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46709: connect: Connection refused (error 111)
I20260620 23:41:25.971627 22873 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46709 } attempt: 1
I20260620 23:41:25.973014 22877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:25.972396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46709 (local address 127.0.0.1:56768)
0620 23:41:25.972655 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:25.972778 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260620 23:41:25.974197 22877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:25.973759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46709 (local address 127.0.0.1:56784)
0620 23:41:25.973935 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:25.974038 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260620 23:41:25.974465 22873 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46709: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46709: connect: Connection refused (error 111)
I20260620 23:41:26.035990 22947 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46709 } attempt: 3
I20260620 23:41:26.037350 22950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:26.036813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46709 (local address 127.0.0.1:56788)
0620 23:41:26.037079 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:26.037177 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260620 23:41:26.038630 22950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:26.038176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46709 (local address 127.0.0.1:56790)
0620 23:41:26.038419 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:26.038500 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
W20260620 23:41:26.038827 22947 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46709: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46709: connect: Connection refused (error 111)
I20260620 23:41:26.042729 22873 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46709 } attempt: 2
I20260620 23:41:26.043926 22877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:26.043434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46709 (local address 127.0.0.1:56800)
0620 23:41:26.043683 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:26.043776 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260620 23:41:26.045073 22877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:26.044655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46709 (local address 127.0.0.1:56816)
0620 23:41:26.044803 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:26.044892 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 23:41:26.045296 22873 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46709: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46709: connect: Connection refused (error 111)
I20260620 23:41:26.151557 22873 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46709 } attempt: 3
I20260620 23:41:26.152841 22877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:26.152264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46709 (local address 127.0.0.1:56824)
0620 23:41:26.152507 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:26.152632 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260620 23:41:26.154052 22877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:26.153591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46709 (local address 127.0.0.1:56838)
0620 23:41:26.153786 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:26.153880 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260620 23:41:26.154241 22873 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46709: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46709: connect: Connection refused (error 111)
I20260620 23:41:26.182045 22947 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46709 } attempt: 4
I20260620 23:41:26.183315 22950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:26.182733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46709 (local address 127.0.0.1:56846)
0620 23:41:26.182999 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:26.183105 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 23:41:26.184340 22950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:26.183950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46709 (local address 127.0.0.1:56854)
0620 23:41:26.184102 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:26.184195 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 23:41:26.184231 22955 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 23:41:26.184556 22947 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46709: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46709: connect: Connection refused (error 111)
W20260620 23:41:26.184700 22955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:41:26.184810 22955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:41:26.184963 22955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:41:26.192709 22955 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:41:26.192804 22955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:41:26.192898 22955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:41:26.192986 22955 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:41:26.193099 22955 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:41:26.203356 22955 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46097,127.0.97.61:34587,127.0.97.60:46709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:46709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:41:26.204742 22955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:41:26.206156 22955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:41:26.218823 22960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:41:26.218781 22961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:41:26.218868 22963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:41:26.219596 22955 server_base.cc:1034] running on GCE node
I20260620 23:41:26.220427 22955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:41:26.221920 22955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:41:26.223136 22955 hybrid_clock.cc:648] HybridClock initialized: now 1781998886223083 us; error 38 us; skew 500 ppm
I20260620 23:41:26.223515 22955 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:41:26 dist-test-slave-tcgx krb5kdc[22793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998886, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:41:26.226713 22955 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260620 23:41:26.228322 22955 webserver.cc:460] Webserver started at http://127.0.97.60:46501/ using document root <none> and password file <none>
I20260620 23:41:26.229050 22955 fs_manager.cc:359] Metadata directory not provided
I20260620 23:41:26.229139 22955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:41:26.229415 22955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:41:26.232532 22955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/minicluster-data/master-2/data/instance:
uuid: "8252ade4a8394d838dd41e4dd95d2b5c"
format_stamp: "Formatted at 2026-06-20 23:41:26 on dist-test-slave-tcgx"
server_key: "6850210e82b321a7fe9371118ac58dcd"
server_key_iv: "35ba29bb250cb563d7c9bfe49b27247b"
server_key_version: "encryptionkey@0"
I20260620 23:41:26.233300 22955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/minicluster-data/master-2/wal/instance:
uuid: "8252ade4a8394d838dd41e4dd95d2b5c"
format_stamp: "Formatted at 2026-06-20 23:41:26 on dist-test-slave-tcgx"
server_key: "6850210e82b321a7fe9371118ac58dcd"
server_key_iv: "35ba29bb250cb563d7c9bfe49b27247b"
server_key_version: "encryptionkey@0"
I20260620 23:41:26.238997 22955 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 23:41:26.242621 22970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:41:26.244346 22955 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 23:41:26.244539 22955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/minicluster-data/master-2/wal
uuid: "8252ade4a8394d838dd41e4dd95d2b5c"
format_stamp: "Formatted at 2026-06-20 23:41:26 on dist-test-slave-tcgx"
server_key: "6850210e82b321a7fe9371118ac58dcd"
server_key_iv: "35ba29bb250cb563d7c9bfe49b27247b"
server_key_version: "encryptionkey@0"
I20260620 23:41:26.244829 22955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:41:26.259802 22955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:41:26.264079 22955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:41:26.264394 22955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:41:26.283666 22955 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:46709
I20260620 23:41:26.283685 23021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:46709 every 8 connection(s)
I20260620 23:41:26.285125 22955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/minicluster-data/master-2/data/info.pb
I20260620 23:41:26.286733 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 22955
I20260620 23:41:26.286918 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998525501163-388-0/minicluster-data/master-2/wal/instance
I20260620 23:41:26.287467 388 external_mini_cluster.cc:1428] Setting key 427a0b24a8990b8dd4b95b3ba0efa7e7
I20260620 23:41:26.290073 23022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:41:26.299162 23022 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46097 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:41:26 dist-test-slave-tcgx krb5kdc[22793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 20 23:41:26 dist-test-slave-tcgx krb5kdc[22793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998886, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:41:26.312441 23025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:26.289487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56860 (local address 127.0.97.60:46709)
0620 23:41:26.290572 (+ 1085us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:26.290590 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:26.290825 (+ 235us) server_negotiation.cc:407] Connection header received
0620 23:41:26.292613 (+ 1788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:26.292635 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:26.293054 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:26.293398 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:26.295299 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:26.296718 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:26.298643 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:26.299457 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:26.299619 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:26.304660 (+ 5041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:41:26.304669 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:41:26.304679 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:41:26.304719 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:41:26.307843 (+ 3124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:26.308573 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:26.308580 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:26.308586 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:26.308750 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:26.309213 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:26.309217 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:26.309219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:26.310010 (+ 791us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:41:26.310319 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0620 23:41:26.310700 (+ 381us) server_negotiation.cc:299] Negotiation successful
0620 23:41:26.311186 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":796,"thread_start_us":422,"threads_started":1}
I20260620 23:41:26.318464 23026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:26.302338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46097 (local address 127.0.0.1:55692)
0620 23:41:26.303181 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:26.303225 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 23:41:26.303377 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:41:26.304259 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:41:26.304264 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:41:26.304947 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:41:26.305276 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:41:26.305285 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:41:26.306431 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:41:26.306439 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:41:26.307751 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:41:26.307764 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:41:26.308556 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:41:26.308563 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:41:26.308737 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:26.309392 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:41:26.309412 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:41:26.311935 (+ 2523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:41:26.315514 (+ 3579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:41:26.315528 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:41:26.315538 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:41:26.316043 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:41:26.316507 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:41:26.316511 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:41:26.316513 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:41:26.316648 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:41:26.317322 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:41:26.317330 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:41:26.317512 (+ 182us) client_negotiation.cc:769] Sending connection context
0620 23:41:26.317693 (+ 181us) client_negotiation.cc:240] Negotiation successful
0620 23:41:26.318010 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":356,"threads_started":1}
I20260620 23:41:26.318483 23027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:26.302425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55692 (local address 127.0.97.62:46097)
0620 23:41:26.303852 (+ 1427us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:26.303860 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:26.303883 (+ 23us) server_negotiation.cc:407] Connection header received
0620 23:41:26.303952 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:26.303960 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:26.304065 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:26.304189 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:26.305430 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:26.306269 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:26.307911 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:26.308336 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:26.308489 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:26.312144 (+ 3655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:41:26.312151 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:41:26.312156 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:41:26.312206 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:41:26.315390 (+ 3184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:26.316178 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:26.316184 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:26.316189 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:26.316386 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:26.316766 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:26.316773 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:26.316779 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:26.317186 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:41:26.317349 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 23:41:26.317816 (+ 467us) server_negotiation.cc:299] Negotiation successful
0620 23:41:26.318027 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":1278,"thread_start_us":1176,"threads_started":1}
I20260620 23:41:26.319901 23022 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34587 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:41:26 dist-test-slave-tcgx krb5kdc[22793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Jun 20 23:41:26 dist-test-slave-tcgx krb5kdc[22793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998886, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 23:41:26.331537 22873 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46709 } attempt: 4
I20260620 23:41:26.333367 23028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:26.315878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55708 (local address 127.0.97.62:46097)
0620 23:41:26.316604 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:26.316608 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:26.316625 (+ 17us) server_negotiation.cc:407] Connection header received
0620 23:41:26.316660 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:26.316663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:26.316747 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:26.316850 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:26.318027 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:26.319427 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:26.320941 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:26.321592 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:26.321684 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:26.327317 (+ 5633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:41:26.327323 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:41:26.327326 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:41:26.327358 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:41:26.330293 (+ 2935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:26.330982 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:26.330990 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:26.330994 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:26.331139 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:26.331523 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:26.331530 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:26.331537 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:26.331923 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:41:26.332206 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0620 23:41:26.332551 (+ 345us) server_negotiation.cc:299] Negotiation successful
0620 23:41:26.332782 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":398,"threads_started":1}
I20260620 23:41:26.336561 22952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:26.320690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50216 (local address 127.0.97.61:34587)
0620 23:41:26.320920 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:26.320926 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:26.321075 (+ 149us) server_negotiation.cc:407] Connection header received
0620 23:41:26.321164 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:26.321170 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:26.321256 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:26.321358 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:26.322282 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:26.323278 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:26.325234 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:26.325797 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:26.325946 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:26.329278 (+ 3332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:41:26.329283 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:41:26.329286 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:41:26.329323 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:41:26.332743 (+ 3420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:26.333668 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:26.333674 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:26.333677 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:26.333833 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:26.334907 (+ 1074us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:26.334914 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:26.334917 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:26.335278 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:41:26.335435 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 23:41:26.335883 (+ 448us) server_negotiation.cc:299] Negotiation successful
0620 23:41:26.336128 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 23:41:26.337379 23026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:26.320712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34587 (local address 127.0.0.1:50216)
0620 23:41:26.320983 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:26.320997 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:41:26.321081 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:41:26.321383 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:41:26.321386 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:41:26.321816 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:41:26.321993 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:41:26.322003 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:41:26.323439 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:41:26.323447 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:41:26.325067 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:41:26.325083 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:41:26.325970 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:41:26.325974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:41:26.326115 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:26.326746 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:41:26.326764 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:41:26.329066 (+ 2302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:41:26.332904 (+ 3838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:41:26.332910 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:41:26.332913 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:41:26.333488 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:41:26.333987 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:41:26.333994 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:41:26.333998 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:41:26.334209 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:41:26.335413 (+ 1204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:41:26.335418 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:41:26.335575 (+ 157us) client_negotiation.cc:769] Sending connection context
0620 23:41:26.336815 (+ 1240us) client_negotiation.cc:240] Negotiation successful
0620 23:41:26.337088 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161}
I20260620 23:41:26.338446 23022 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46709 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:41:26 dist-test-slave-tcgx krb5kdc[22793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 20 23:41:26 dist-test-slave-tcgx krb5kdc[22793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Jun 20 23:41:26 dist-test-slave-tcgx krb5kdc[22793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998886, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260620 23:41:26.348498 22877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:26.332173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46709 (local address 127.0.0.1:56874)
0620 23:41:26.332365 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:26.332386 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:41:26.332486 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:41:26.333128 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:41:26.333133 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:41:26.333628 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:41:26.333800 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:41:26.333809 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:41:26.334931 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:41:26.334936 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:41:26.336334 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:41:26.336351 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:41:26.337335 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:41:26.337341 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:41:26.337857 (+ 516us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:26.338446 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:41:26.338464 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:41:26.341313 (+ 2849us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:41:26.345738 (+ 4425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:41:26.345744 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:41:26.345748 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:41:26.346173 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:41:26.346577 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:41:26.346582 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:41:26.346585 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:41:26.346741 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:41:26.347405 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:41:26.347412 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:41:26.347603 (+ 191us) client_negotiation.cc:769] Sending connection context
0620 23:41:26.347768 (+ 165us) client_negotiation.cc:240] Negotiation successful
0620 23:41:26.348072 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 23:41:26.349630 23025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:26.332639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56874 (local address 127.0.97.60:46709)
0620 23:41:26.332809 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:26.332815 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:26.332832 (+ 17us) server_negotiation.cc:407] Connection header received
0620 23:41:26.332881 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:26.332886 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:26.332962 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:26.333076 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:26.333935 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:26.334792 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:26.336485 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:26.337174 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:26.337305 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:26.341505 (+ 4200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:41:26.341512 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:41:26.341517 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:41:26.341557 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:41:26.345583 (+ 4026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:26.346306 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:26.346312 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:26.346315 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:26.346461 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:26.346856 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:26.346861 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:26.346865 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:26.347218 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:41:26.347429 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 23:41:26.348963 (+ 1534us) server_negotiation.cc:299] Negotiation successful
0620 23:41:26.349218 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 23:41:26.354238 23029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:26.335872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50228 (local address 127.0.97.61:34587)
0620 23:41:26.336493 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:26.336499 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:26.336524 (+ 25us) server_negotiation.cc:407] Connection header received
0620 23:41:26.336573 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:26.336578 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:26.336643 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:26.338250 (+ 1607us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:26.338551 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:26.339712 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:26.341489 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:26.341924 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:26.342064 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:26.347432 (+ 5368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:41:26.347441 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:41:26.347449 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:41:26.347487 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:41:26.351149 (+ 3662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:26.352013 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:26.352019 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:26.352022 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:26.352184 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:26.352651 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:26.352657 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:26.352663 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:26.353026 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:41:26.353290 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0620 23:41:26.353584 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 23:41:26.353812 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":284,"threads_started":1}
I20260620 23:41:26.355651 23026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:26.339193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46709 (local address 127.0.0.1:56890)
0620 23:41:26.339356 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:26.339374 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:41:26.339464 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:41:26.340657 (+ 1193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:41:26.340668 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:41:26.341047 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:41:26.341170 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:41:26.341177 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:41:26.342843 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:41:26.342850 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:41:26.344115 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:41:26.344127 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:41:26.344929 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:41:26.344934 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:41:26.345119 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:26.345689 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:41:26.345710 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:41:26.348417 (+ 2707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:41:26.352939 (+ 4522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:41:26.352944 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:41:26.352948 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:41:26.353439 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:41:26.353827 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:41:26.353831 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:41:26.353835 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:41:26.353963 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:41:26.354600 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:41:26.354607 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:41:26.354760 (+ 153us) client_negotiation.cc:769] Sending connection context
0620 23:41:26.354912 (+ 152us) client_negotiation.cc:240] Negotiation successful
0620 23:41:26.355277 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 23:41:26.356204 23030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:26.339395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56890 (local address 127.0.97.60:46709)
0620 23:41:26.340336 (+ 941us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:26.340340 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:26.340357 (+ 17us) server_negotiation.cc:407] Connection header received
0620 23:41:26.340392 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:26.340395 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:26.340441 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:26.340566 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:26.341318 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:26.342687 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:26.344242 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:26.344766 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:26.344911 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:26.348829 (+ 3918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:41:26.348835 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:41:26.348840 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:41:26.348876 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:41:26.352809 (+ 3933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:26.353571 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:26.353578 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:26.353580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:26.353717 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:26.354077 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:26.354081 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:26.354085 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:26.354412 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:41:26.354547 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 23:41:26.355066 (+ 519us) server_negotiation.cc:299] Negotiation successful
0620 23:41:26.355444 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":143,"server-negotiator.queue_time_us":851,"spinlock_wait_cycles":30976,"thread_start_us":692,"threads_started":1}
I20260620 23:41:26.356217 22873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad1d7e3982a542e4a95b44470e9af1aa: Bootstrap starting.
I20260620 23:41:26.359423 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:41:26.360149 22873 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad1d7e3982a542e4a95b44470e9af1aa: Neither blocks nor log segments found. Creating new log.
I20260620 23:41:26.361231 22873 log.cc:824] T 00000000000000000000000000000000 P ad1d7e3982a542e4a95b44470e9af1aa: Log is configured to *not* fsync() on all Append() calls
I20260620 23:41:26.362020 23022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8252ade4a8394d838dd41e4dd95d2b5c: Bootstrap starting.
I20260620 23:41:26.364332 22873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad1d7e3982a542e4a95b44470e9af1aa: No bootstrap required, opened a new log
I20260620 23:41:26.366066 23022 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8252ade4a8394d838dd41e4dd95d2b5c: Neither blocks nor log segments found. Creating new log.
I20260620 23:41:26.362928 22848 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "ad1d7e3982a542e4a95b44470e9af1aa"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55708
I20260620 23:41:26.367408 23022 log.cc:824] T 00000000000000000000000000000000 P 8252ade4a8394d838dd41e4dd95d2b5c: Log is configured to *not* fsync() on all Append() calls
I20260620 23:41:26.370697 23022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8252ade4a8394d838dd41e4dd95d2b5c: No bootstrap required, opened a new log
I20260620 23:41:26.371701 22873 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad1d7e3982a542e4a95b44470e9af1aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad1d7e3982a542e4a95b44470e9af1aa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46097 } } peers { permanent_uuid: "f2a4ad556e23425ebf8c2f9d0605a501" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34587 } } peers { permanent_uuid: "8252ade4a8394d838dd41e4dd95d2b5c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46709 } }
I20260620 23:41:26.372061 22873 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad1d7e3982a542e4a95b44470e9af1aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad1d7e3982a542e4a95b44470e9af1aa, State: Initialized, Role: FOLLOWER
I20260620 23:41:26.372503 22873 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad1d7e3982a542e4a95b44470e9af1aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad1d7e3982a542e4a95b44470e9af1aa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46097 } } peers { permanent_uuid: "f2a4ad556e23425ebf8c2f9d0605a501" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34587 } } peers { permanent_uuid: "8252ade4a8394d838dd41e4dd95d2b5c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46709 } }
I20260620 23:41:26.373435 23033 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad1d7e3982a542e4a95b44470e9af1aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad1d7e3982a542e4a95b44470e9af1aa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46097 } } peers { permanent_uuid: "f2a4ad556e23425ebf8c2f9d0605a501" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34587 } } peers { permanent_uuid: "8252ade4a8394d838dd41e4dd95d2b5c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46709 } } }
I20260620 23:41:26.373818 23033 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad1d7e3982a542e4a95b44470e9af1aa [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:41:26.374583 22873 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad1d7e3982a542e4a95b44470e9af1aa [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:41:26.374716 22848 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad1d7e3982a542e4a95b44470e9af1aa [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 23:41:26.374965 22848 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad1d7e3982a542e4a95b44470e9af1aa [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:41:26.378386 22848 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad1d7e3982a542e4a95b44470e9af1aa [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad1d7e3982a542e4a95b44470e9af1aa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46097 } } peers { permanent_uuid: "f2a4ad556e23425ebf8c2f9d0605a501" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34587 } } peers { permanent_uuid: "8252ade4a8394d838dd41e4dd95d2b5c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46709 } }
I20260620 23:41:26.381600 22848 leader_election.cc:288] T 00000000000000000000000000000000 P ad1d7e3982a542e4a95b44470e9af1aa [CANDIDATE]: Term 1 election: Requested vote from peers f2a4ad556e23425ebf8c2f9d0605a501 (127.0.97.61:34587), 8252ade4a8394d838dd41e4dd95d2b5c (127.0.97.60:46709)
I20260620 23:41:26.381604 23022 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8252ade4a8394d838dd41e4dd95d2b5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad1d7e3982a542e4a95b44470e9af1aa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46097 } } peers { permanent_uuid: "f2a4ad556e23425ebf8c2f9d0605a501" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34587 } } peers { permanent_uuid: "8252ade4a8394d838dd41e4dd95d2b5c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46709 } }
I20260620 23:41:26.382076 23022 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8252ade4a8394d838dd41e4dd95d2b5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8252ade4a8394d838dd41e4dd95d2b5c, State: Initialized, Role: FOLLOWER
I20260620 23:41:26.382611 23022 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8252ade4a8394d838dd41e4dd95d2b5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad1d7e3982a542e4a95b44470e9af1aa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46097 } } peers { permanent_uuid: "f2a4ad556e23425ebf8c2f9d0605a501" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34587 } } peers { permanent_uuid: "8252ade4a8394d838dd41e4dd95d2b5c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46709 } }
I20260620 23:41:26.385519 23022 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8252ade4a8394d838dd41e4dd95d2b5c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:41:26.385622 22997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ad1d7e3982a542e4a95b44470e9af1aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8252ade4a8394d838dd41e4dd95d2b5c"
I20260620 23:41:26.386224 22997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8252ade4a8394d838dd41e4dd95d2b5c [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:41:26.391156 22997 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8252ade4a8394d838dd41e4dd95d2b5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad1d7e3982a542e4a95b44470e9af1aa in term 1.
I20260620 23:41:26.384434 23042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8252ade4a8394d838dd41e4dd95d2b5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad1d7e3982a542e4a95b44470e9af1aa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46097 } } peers { permanent_uuid: "f2a4ad556e23425ebf8c2f9d0605a501" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34587 } } peers { permanent_uuid: "8252ade4a8394d838dd41e4dd95d2b5c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46709 } } }
I20260620 23:41:26.392411 23042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8252ade4a8394d838dd41e4dd95d2b5c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:41:26.392565 22822 leader_election.cc:302] T 00000000000000000000000000000000 P ad1d7e3982a542e4a95b44470e9af1aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8252ade4a8394d838dd41e4dd95d2b5c, ad1d7e3982a542e4a95b44470e9af1aa; no voters:
I20260620 23:41:26.393270 23033 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad1d7e3982a542e4a95b44470e9af1aa [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:41:26.394258 23033 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad1d7e3982a542e4a95b44470e9af1aa [term 1 LEADER]: Becoming Leader. State: Replica: ad1d7e3982a542e4a95b44470e9af1aa, State: Running, Role: LEADER
I20260620 23:41:26.395083 23033 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad1d7e3982a542e4a95b44470e9af1aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad1d7e3982a542e4a95b44470e9af1aa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46097 } } peers { permanent_uuid: "f2a4ad556e23425ebf8c2f9d0605a501" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34587 } } peers { permanent_uuid: "8252ade4a8394d838dd41e4dd95d2b5c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46709 } }
I20260620 23:41:26.402344 23047 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad1d7e3982a542e4a95b44470e9af1aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad1d7e3982a542e4a95b44470e9af1aa. Latest consensus state: current_term: 1 leader_uuid: "ad1d7e3982a542e4a95b44470e9af1aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad1d7e3982a542e4a95b44470e9af1aa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46097 } } peers { permanent_uuid: "f2a4ad556e23425ebf8c2f9d0605a501" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34587 } } peers { permanent_uuid: "8252ade4a8394d838dd41e4dd95d2b5c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46709 } } }
I20260620 23:41:26.403578 23047 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad1d7e3982a542e4a95b44470e9af1aa [sys.catalog]: This master's current role is: LEADER
I20260620 23:41:26.412915 23053 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 23:41:26.414237 23055 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:41:26.420253 23055 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260620 23:41:26.421921 22827 tablet.cc:2369] T 00000000000000000000000000000000 P ad1d7e3982a542e4a95b44470e9af1aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:41:26.418833 22806 master_runner.cc:186] Error getting master registration for 127.0.97.61:34587: OK, instance_id { permanent_uuid: "f2a4ad556e23425ebf8c2f9d0605a501" instance_seqno: 1781998885750736 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 23:41:26.431861 22806 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 23:41:26.442456 22997 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8252ade4a8394d838dd41e4dd95d2b5c [term 1 FOLLOWER]: Refusing update from remote peer ad1d7e3982a542e4a95b44470e9af1aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:41:26.443735 23047 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ad1d7e3982a542e4a95b44470e9af1aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "8252ade4a8394d838dd41e4dd95d2b5c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:41:26.468379 23042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8252ade4a8394d838dd41e4dd95d2b5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad1d7e3982a542e4a95b44470e9af1aa. Latest consensus state: current_term: 1 leader_uuid: "ad1d7e3982a542e4a95b44470e9af1aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad1d7e3982a542e4a95b44470e9af1aa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46097 } } peers { permanent_uuid: "f2a4ad556e23425ebf8c2f9d0605a501" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34587 } } peers { permanent_uuid: "8252ade4a8394d838dd41e4dd95d2b5c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46709 } } }
I20260620 23:41:26.468703 23042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8252ade4a8394d838dd41e4dd95d2b5c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:41:26.469712 23065 mvcc.cc:205] Tried to move back new op lower bound from 7299067438856499200 to 7299067438684258304. Current Snapshot: MvccSnapshot[applied={T|T < 7299067438856499200}]
I20260620 23:41:26.471201 23029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:26.426201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50236 (local address 127.0.97.61:34587)
0620 23:41:26.426774 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:26.426785 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:26.439517 (+ 12732us) server_negotiation.cc:407] Connection header received
0620 23:41:26.441008 (+ 1491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:26.441018 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:26.441118 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:26.441202 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:26.445668 (+ 4466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:26.446722 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:26.454200 (+ 7478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:26.455096 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:26.455209 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:26.461868 (+ 6659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:41:26.461879 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:41:26.461884 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:41:26.461936 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:41:26.465621 (+ 3685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:26.466476 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:26.466482 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:26.466485 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:26.466707 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:26.469174 (+ 2467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:26.469181 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:26.469188 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:26.469648 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:41:26.469914 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0620 23:41:26.470460 (+ 546us) server_negotiation.cc:299] Negotiation successful
0620 23:41:26.470754 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373}
I20260620 23:41:26.479282 23033 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad1d7e3982a542e4a95b44470e9af1aa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ad1d7e3982a542e4a95b44470e9af1aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad1d7e3982a542e4a95b44470e9af1aa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46097 } } peers { permanent_uuid: "f2a4ad556e23425ebf8c2f9d0605a501" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34587 } } peers { permanent_uuid: "8252ade4a8394d838dd41e4dd95d2b5c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46709 } } }
I20260620 23:41:26.479692 23033 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad1d7e3982a542e4a95b44470e9af1aa [sys.catalog]: This master's current role is: LEADER
I20260620 23:41:26.481679 23055 catalog_manager.cc:1347] Generated new cluster ID: fe9f61d1e9b44f65a1a25a62fc9eac29
I20260620 23:41:26.481779 23055 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:41:26.485193 23030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:26.440093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56894 (local address 127.0.97.60:46709)
0620 23:41:26.440317 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:26.440325 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:26.450428 (+ 10103us) server_negotiation.cc:407] Connection header received
0620 23:41:26.451406 (+ 978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:26.451414 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:26.451517 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:26.451617 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:26.458058 (+ 6441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:26.459198 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:26.467206 (+ 8008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:26.467970 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:26.468091 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:26.476425 (+ 8334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:41:26.476434 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:41:26.476441 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:41:26.476499 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:41:26.480946 (+ 4447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:26.482205 (+ 1259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:26.482215 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:26.482220 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:26.482431 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:26.482999 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:26.483007 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:26.483012 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:26.483526 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:41:26.483876 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0620 23:41:26.484518 (+ 642us) server_negotiation.cc:299] Negotiation successful
0620 23:41:26.484819 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 23:41:26.486007 23028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:26.426995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55714 (local address 127.0.97.62:46097)
0620 23:41:26.427323 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:26.427331 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:26.439376 (+ 12045us) server_negotiation.cc:407] Connection header received
0620 23:41:26.439977 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:26.439983 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:26.440083 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:26.440164 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:26.443137 (+ 2973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:26.444564 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:26.448728 (+ 4164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:26.449649 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:26.449792 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:26.458973 (+ 9181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:41:26.458983 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:41:26.458988 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:41:26.459043 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:41:26.463150 (+ 4107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:26.464606 (+ 1456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:26.464613 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:26.464617 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:26.464768 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:26.465497 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:26.465505 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:26.465512 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:26.465953 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:41:26.466255 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0620 23:41:26.485243 (+ 18988us) server_negotiation.cc:299] Negotiation successful
0620 23:41:26.485576 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260620 23:41:26.487140 22947 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46709 } attempt: 5
I20260620 23:41:26.487799 23068 mvcc.cc:205] Tried to move back new op lower bound from 7299067438856499200 to 7299067438684258304. Current Snapshot: MvccSnapshot[applied={T|T < 7299067438856499200}]
I20260620 23:41:26.488591 23042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8252ade4a8394d838dd41e4dd95d2b5c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ad1d7e3982a542e4a95b44470e9af1aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad1d7e3982a542e4a95b44470e9af1aa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46097 } } peers { permanent_uuid: "f2a4ad556e23425ebf8c2f9d0605a501" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34587 } } peers { permanent_uuid: "8252ade4a8394d838dd41e4dd95d2b5c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46709 } } }
I20260620 23:41:26.488852 23042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8252ade4a8394d838dd41e4dd95d2b5c [sys.catalog]: This master's current role is: FOLLOWER
Jun 20 23:41:26 dist-test-slave-tcgx krb5kdc[22793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260620 23:41:26.511869 23070 catalog_manager.cc:1259] Loaded cluster ID: fe9f61d1e9b44f65a1a25a62fc9eac29
I20260620 23:41:26.511979 23070 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8252ade4a8394d838dd41e4dd95d2b5c: loading cluster ID for follower catalog manager: success
W20260620 23:41:26.513300 23070 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 8252ade4a8394d838dd41e4dd95d2b5c: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260620 23:41:26.513473 23070 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:41:26.499909 22955 master_runner.cc:186] Error getting master registration for 127.0.97.61:34587: OK, instance_id { permanent_uuid: "f2a4ad556e23425ebf8c2f9d0605a501" instance_seqno: 1781998885750736 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 23:41:26.526979 22955 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 23:41:26.536765 22950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:26.487762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46709 (local address 127.0.0.1:56898)
0620 23:41:26.488158 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:26.488178 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:41:26.488295 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:41:26.489190 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:41:26.489195 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:41:26.489704 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:41:26.489922 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:41:26.489936 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:41:26.491490 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:41:26.491495 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:41:26.492934 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:41:26.492950 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:41:26.494705 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:41:26.494711 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:41:26.494910 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:26.499500 (+ 4590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:41:26.499525 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:41:26.507831 (+ 8306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:41:26.533523 (+ 25692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:41:26.533531 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:41:26.533536 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:41:26.534177 (+ 641us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:41:26.534729 (+ 552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:41:26.534735 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:41:26.534739 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:41:26.534955 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:41:26.535689 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:41:26.535694 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:41:26.535878 (+ 184us) client_negotiation.cc:769] Sending connection context
0620 23:41:26.536039 (+ 161us) client_negotiation.cc:240] Negotiation successful
0620 23:41:26.536354 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257}
I20260620 23:41:26.537367 23030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:26.488512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56898 (local address 127.0.97.60:46709)
0620 23:41:26.488681 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:26.488688 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:26.488705 (+ 17us) server_negotiation.cc:407] Connection header received
0620 23:41:26.488758 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:26.488762 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:26.488868 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:26.488980 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:26.490143 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:26.491319 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:26.493622 (+ 2303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:26.494330 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:26.494455 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:26.508162 (+ 13707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:41:26.508170 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:41:26.508175 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:41:26.508262 (+ 87us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:41:26.532768 (+ 24506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:26.534355 (+ 1587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:26.534361 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:26.534365 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:26.534535 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:26.535127 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:26.535132 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:26.535135 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:26.535540 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:41:26.535756 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 23:41:26.536248 (+ 492us) server_negotiation.cc:299] Negotiation successful
0620 23:41:26.536687 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 23:41:26.543622 22947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2a4ad556e23425ebf8c2f9d0605a501: Bootstrap starting.
I20260620 23:41:26.543684 23055 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:41:26.545711 23055 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:41:26.547956 22947 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2a4ad556e23425ebf8c2f9d0605a501: Neither blocks nor log segments found. Creating new log.
I20260620 23:41:26.549430 22947 log.cc:824] T 00000000000000000000000000000000 P f2a4ad556e23425ebf8c2f9d0605a501: Log is configured to *not* fsync() on all Append() calls
I20260620 23:41:26.556604 22947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2a4ad556e23425ebf8c2f9d0605a501: No bootstrap required, opened a new log
I20260620 23:41:26.562194 23055 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad1d7e3982a542e4a95b44470e9af1aa: Generated new TSK 0
I20260620 23:41:26.563408 23055 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:41:26.568409 22947 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2a4ad556e23425ebf8c2f9d0605a501 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad1d7e3982a542e4a95b44470e9af1aa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46097 } } peers { permanent_uuid: "f2a4ad556e23425ebf8c2f9d0605a501" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34587 } } peers { permanent_uuid: "8252ade4a8394d838dd41e4dd95d2b5c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46709 } }
I20260620 23:41:26.568857 22947 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2a4ad556e23425ebf8c2f9d0605a501 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2a4ad556e23425ebf8c2f9d0605a501, State: Initialized, Role: FOLLOWER
I20260620 23:41:26.569432 22947 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2a4ad556e23425ebf8c2f9d0605a501 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad1d7e3982a542e4a95b44470e9af1aa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46097 } } peers { permanent_uuid: "f2a4ad556e23425ebf8c2f9d0605a501" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34587 } } peers { permanent_uuid: "8252ade4a8394d838dd41e4dd95d2b5c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46709 } }
I20260620 23:41:26.570428 23075 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2a4ad556e23425ebf8c2f9d0605a501 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad1d7e3982a542e4a95b44470e9af1aa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46097 } } peers { permanent_uuid: "f2a4ad556e23425ebf8c2f9d0605a501" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34587 } } peers { permanent_uuid: "8252ade4a8394d838dd41e4dd95d2b5c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46709 } } }
I20260620 23:41:26.570868 23075 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2a4ad556e23425ebf8c2f9d0605a501 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:41:26.571880 22947 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2a4ad556e23425ebf8c2f9d0605a501 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:41:26.571938 22922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ad1d7e3982a542e4a95b44470e9af1aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f2a4ad556e23425ebf8c2f9d0605a501"
I20260620 23:41:26.572314 22921 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2a4ad556e23425ebf8c2f9d0605a501 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:41:26.575707 22921 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f2a4ad556e23425ebf8c2f9d0605a501 [term 1 FOLLOWER]: Refusing update from remote peer ad1d7e3982a542e4a95b44470e9af1aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:41:26.576795 23047 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ad1d7e3982a542e4a95b44470e9af1aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2a4ad556e23425ebf8c2f9d0605a501" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:41:26.578827 22922 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f2a4ad556e23425ebf8c2f9d0605a501 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad1d7e3982a542e4a95b44470e9af1aa in term 1.
I20260620 23:41:26.608234 23075 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2a4ad556e23425ebf8c2f9d0605a501 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad1d7e3982a542e4a95b44470e9af1aa. Latest consensus state: current_term: 1 leader_uuid: "ad1d7e3982a542e4a95b44470e9af1aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad1d7e3982a542e4a95b44470e9af1aa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46097 } } peers { permanent_uuid: "f2a4ad556e23425ebf8c2f9d0605a501" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34587 } } peers { permanent_uuid: "8252ade4a8394d838dd41e4dd95d2b5c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46709 } } }
I20260620 23:41:26.608592 23075 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2a4ad556e23425ebf8c2f9d0605a501 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:41:26.608982 23084 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2a4ad556e23425ebf8c2f9d0605a501 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ad1d7e3982a542e4a95b44470e9af1aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad1d7e3982a542e4a95b44470e9af1aa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46097 } } peers { permanent_uuid: "f2a4ad556e23425ebf8c2f9d0605a501" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34587 } } peers { permanent_uuid: "8252ade4a8394d838dd41e4dd95d2b5c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46709 } } }
I20260620 23:41:26.609328 23084 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2a4ad556e23425ebf8c2f9d0605a501 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:41:26.662817 23093 catalog_manager.cc:1259] Loaded cluster ID: fe9f61d1e9b44f65a1a25a62fc9eac29
I20260620 23:41:26.663048 23093 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f2a4ad556e23425ebf8c2f9d0605a501: loading cluster ID for follower catalog manager: success
I20260620 23:41:26.667119 23093 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f2a4ad556e23425ebf8c2f9d0605a501: acquiring CA information for follower catalog manager: success
I20260620 23:41:26.675551 23093 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f2a4ad556e23425ebf8c2f9d0605a501: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 23:41:26.755856 23028 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:41:26.734331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55724 (local address 127.0.97.62:46097)
0620 23:41:26.734839 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:26.734846 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:26.753491 (+ 18645us) server_negotiation.cc:407] Connection header received
0620 23:41:26.754048 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:26.754055 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:26.754148 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:26.754226 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:26.755540 (+ 1314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55724: BlockingRecv error: recv got EOF from 127.0.0.1:55724 (error 108)
Metrics: {"server-negotiator.queue_time_us":357}
W20260620 23:41:26.759754 23029 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:41:26.734628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50244 (local address 127.0.97.61:34587)
0620 23:41:26.734994 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:26.735000 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:26.756982 (+ 21982us) server_negotiation.cc:407] Connection header received
0620 23:41:26.758104 (+ 1122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:26.758111 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:26.758202 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:26.758284 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:26.759494 (+ 1210us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50244: BlockingRecv error: recv got EOF from 127.0.0.1:50244 (error 108)
Metrics: {"server-negotiator.queue_time_us":238}
W20260620 23:41:26.768321 23030 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:41:26.757160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56912 (local address 127.0.97.60:46709)
0620 23:41:26.757358 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:26.757364 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:26.757729 (+ 365us) server_negotiation.cc:407] Connection header received
0620 23:41:26.757914 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:26.757920 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:26.758008 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:26.758107 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:26.768028 (+ 9921us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56912: BlockingRecv error: recv got EOF from 127.0.0.1:56912 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 23:41:26.823267 23029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:26.802863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50246 (local address 127.0.97.61:34587)
0620 23:41:26.803129 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:26.803137 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:26.813950 (+ 10813us) server_negotiation.cc:407] Connection header received
0620 23:41:26.814597 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:26.814604 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:26.814869 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:26.815060 (+ 191us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:41:26.815903 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:26.817252 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:26.819631 (+ 2379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:26.820431 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:26.820546 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:26.821450 (+ 904us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:41:26.822280 (+ 830us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:41:26.822424 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 23:41:26.822698 (+ 274us) server_negotiation.cc:299] Negotiation successful
0620 23:41:26.822811 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
W20260620 23:41:26.878695 23030 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:41:26.865144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56920 (local address 127.0.97.60:46709)
0620 23:41:26.865366 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:26.865373 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:26.876317 (+ 10944us) server_negotiation.cc:407] Connection header received
0620 23:41:26.876945 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:26.876952 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:26.877110 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:26.877201 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:26.878411 (+ 1210us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56920: BlockingRecv error: recv got EOF from 127.0.0.1:56920 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 23:41:26.963587 23029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:26.941314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50254 (local address 127.0.97.61:34587)
0620 23:41:26.941511 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:26.941518 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:26.955703 (+ 14185us) server_negotiation.cc:407] Connection header received
0620 23:41:26.956079 (+ 376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:26.956087 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:26.956195 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:26.956273 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:41:26.957111 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:26.958421 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:26.960853 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:26.961657 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:26.961787 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:26.962534 (+ 747us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:41:26.962700 (+ 166us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:41:26.962774 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 23:41:26.963078 (+ 304us) server_negotiation.cc:299] Negotiation successful
0620 23:41:26.963181 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260620 23:41:27.003028 23030 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:41:27.001173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56924 (local address 127.0.97.60:46709)
0620 23:41:27.001390 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:27.001396 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:27.001422 (+ 26us) server_negotiation.cc:407] Connection header received
0620 23:41:27.001510 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:27.001519 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:27.001637 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:27.001764 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:27.002703 (+ 939us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56924: BlockingRecv error: recv got EOF from 127.0.0.1:56924 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 23:41:27.057785 23029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:27.039746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50266 (local address 127.0.97.61:34587)
0620 23:41:27.039974 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:27.039982 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:27.050661 (+ 10679us) server_negotiation.cc:407] Connection header received
0620 23:41:27.051219 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:27.051227 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:27.051326 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:27.051415 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:41:27.052142 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:27.053323 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:27.055370 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:27.056046 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:27.056155 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:27.056855 (+ 700us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:41:27.057018 (+ 163us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:41:27.057092 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 23:41:27.057342 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 23:41:27.057452 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260620 23:41:27.109702 23030 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:41:27.092186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56938 (local address 127.0.97.60:46709)
0620 23:41:27.092431 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:27.092441 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:27.107659 (+ 15218us) server_negotiation.cc:407] Connection header received
0620 23:41:27.108036 (+ 377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:27.108045 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:27.108150 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:27.108248 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:27.109444 (+ 1196us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56938: BlockingRecv error: recv got EOF from 127.0.0.1:56938 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 23:41:27.155391 23029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:27.147828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50270 (local address 127.0.97.61:34587)
0620 23:41:27.148027 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:27.148034 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:27.148680 (+ 646us) server_negotiation.cc:407] Connection header received
0620 23:41:27.149011 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:27.149019 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:27.149142 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:27.149259 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:41:27.149829 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:27.151097 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:27.153145 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:27.153788 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:27.153926 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:27.154470 (+ 544us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:41:27.154632 (+ 162us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:41:27.154735 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 23:41:27.154966 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 23:41:27.155072 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260620 23:41:27.165443 23030 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:41:27.162785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56948 (local address 127.0.97.60:46709)
0620 23:41:27.163076 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:27.163083 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:27.163650 (+ 567us) server_negotiation.cc:407] Connection header received
0620 23:41:27.163885 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:27.163892 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:27.163996 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:27.164143 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:27.165192 (+ 1049us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56948: BlockingRecv error: recv got EOF from 127.0.0.1:56948 (error 108)
Metrics: {"server-negotiator.queue_time_us":131}
I20260620 23:41:27.185079 23029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:27.177435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50286 (local address 127.0.97.61:34587)
0620 23:41:27.177646 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:27.177653 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:27.178232 (+ 579us) server_negotiation.cc:407] Connection header received
0620 23:41:27.178466 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:27.178472 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:27.178583 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:27.178697 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:41:27.179257 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:27.180490 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:27.182594 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:27.183420 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:27.183577 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:27.184118 (+ 541us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:41:27.184276 (+ 158us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:41:27.184385 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 23:41:27.184578 (+ 193us) server_negotiation.cc:299] Negotiation successful
0620 23:41:27.184706 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 23:41:27.212774 23047 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad1d7e3982a542e4a95b44470e9af1aa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ad1d7e3982a542e4a95b44470e9af1aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad1d7e3982a542e4a95b44470e9af1aa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46097 } } peers { permanent_uuid: "f2a4ad556e23425ebf8c2f9d0605a501" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34587 } } peers { permanent_uuid: "8252ade4a8394d838dd41e4dd95d2b5c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46709 } } }
I20260620 23:41:27.213270 23047 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad1d7e3982a542e4a95b44470e9af1aa [sys.catalog]: This master's current role is: LEADER
W20260620 23:41:27.214497 23030 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:41:27.209336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56954 (local address 127.0.97.60:46709)
0620 23:41:27.209514 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:27.209521 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:27.211991 (+ 2470us) server_negotiation.cc:407] Connection header received
0620 23:41:27.212644 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:27.212649 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:27.212752 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:27.212834 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:27.214089 (+ 1255us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56954: BlockingRecv error: recv got EOF from 127.0.0.1:56954 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 23:41:27.242240 23029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:27.236070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50288 (local address 127.0.97.61:34587)
0620 23:41:27.236325 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:27.236335 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:27.236909 (+ 574us) server_negotiation.cc:407] Connection header received
0620 23:41:27.237159 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:27.237163 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:27.237236 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:27.237342 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:41:27.237780 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:27.238601 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:27.240121 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:27.240721 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:27.240865 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:27.241407 (+ 542us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:41:27.241542 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:41:27.241620 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 23:41:27.241830 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 23:41:27.241910 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
W20260620 23:41:27.255018 23030 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:41:27.252731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56962 (local address 127.0.97.60:46709)
0620 23:41:27.252993 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:27.253002 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:27.253555 (+ 553us) server_negotiation.cc:407] Connection header received
0620 23:41:27.253784 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:27.253790 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:27.253878 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:27.253993 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:27.254742 (+ 749us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56962: BlockingRecv error: recv got EOF from 127.0.0.1:56962 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
I20260620 23:41:27.296876 23029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:27.290826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50290 (local address 127.0.97.61:34587)
0620 23:41:27.291126 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:27.291134 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:27.291499 (+ 365us) server_negotiation.cc:407] Connection header received
0620 23:41:27.291698 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:27.291702 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:27.291775 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:27.291861 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:41:27.292316 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:27.293086 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:27.295088 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:27.295506 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:27.295601 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:27.296122 (+ 521us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:41:27.296246 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:41:27.296316 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 23:41:27.296483 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 23:41:27.296555 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260620 23:41:27.305253 23030 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:41:27.303134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56974 (local address 127.0.97.60:46709)
0620 23:41:27.303333 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:27.303338 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:27.303775 (+ 437us) server_negotiation.cc:407] Connection header received
0620 23:41:27.304001 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:27.304005 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:27.304084 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:27.304191 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:27.304974 (+ 783us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56974: BlockingRecv error: recv got EOF from 127.0.0.1:56974 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 23:41:27.378595 23029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:27.373622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50300 (local address 127.0.97.61:34587)
0620 23:41:27.373847 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:27.373856 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:27.374081 (+ 225us) server_negotiation.cc:407] Connection header received
0620 23:41:27.374244 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:27.374249 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:27.374316 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:27.374399 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:41:27.374790 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:27.375686 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:27.377018 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:27.377414 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:27.377509 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:27.377868 (+ 359us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:41:27.377983 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:41:27.378048 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 23:41:27.378188 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 23:41:27.378253 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260620 23:41:27.386368 23030 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:41:27.384520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56984 (local address 127.0.97.60:46709)
0620 23:41:27.384775 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:27.384782 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:27.385028 (+ 246us) server_negotiation.cc:407] Connection header received
0620 23:41:27.385180 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:27.385184 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:27.385256 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:27.385341 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:27.386119 (+ 778us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56984: BlockingRecv error: recv got EOF from 127.0.0.1:56984 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 23:41:27.517725 23070 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8252ade4a8394d838dd41e4dd95d2b5c: acquiring CA information for follower catalog manager: success
I20260620 23:41:27.520452 23070 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8252ade4a8394d838dd41e4dd95d2b5c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 23:41:27.524121 23029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:27.518225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50306 (local address 127.0.97.61:34587)
0620 23:41:27.518411 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:27.518416 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:27.519486 (+ 1070us) server_negotiation.cc:407] Connection header received
0620 23:41:27.519720 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:27.519726 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:27.519808 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:27.519910 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:41:27.520320 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:27.521121 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:27.522416 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:27.522857 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:27.523038 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:27.523413 (+ 375us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:41:27.523517 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:41:27.523623 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 23:41:27.523769 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 23:41:27.523851 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 23:41:27.535531 23030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:27.530081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56988 (local address 127.0.97.60:46709)
0620 23:41:27.530328 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:27.530334 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:27.530556 (+ 222us) server_negotiation.cc:407] Connection header received
0620 23:41:27.530755 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:27.530759 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:27.530915 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:27.531069 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:41:27.531430 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:27.532158 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:27.533404 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:27.533792 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:27.533882 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:27.534335 (+ 453us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:41:27.534881 (+ 546us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:41:27.535076 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0620 23:41:27.535117 (+ 41us) server_negotiation.cc:299] Negotiation successful
0620 23:41:27.535185 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260620 23:41:27.549435 23029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:27.541805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50320 (local address 127.0.97.61:34587)
0620 23:41:27.542077 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:27.542087 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:27.542718 (+ 631us) server_negotiation.cc:407] Connection header received
0620 23:41:27.542916 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:27.542924 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:27.543066 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:27.543158 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:41:27.544031 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:27.545083 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:27.547295 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:27.547939 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:27.548071 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:27.548556 (+ 485us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:41:27.548723 (+ 167us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:41:27.548815 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 23:41:27.548978 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 23:41:27.549073 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 23:41:27.551097 23030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:27.543110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57002 (local address 127.0.97.60:46709)
0620 23:41:27.543387 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:27.543395 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:27.544170 (+ 775us) server_negotiation.cc:407] Connection header received
0620 23:41:27.544403 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:27.544408 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:27.544512 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:27.544610 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:41:27.545075 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:27.545876 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:27.548170 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:27.548855 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:27.548990 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:27.550174 (+ 1184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:41:27.550313 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:41:27.550409 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 23:41:27.550725 (+ 316us) server_negotiation.cc:299] Negotiation successful
0620 23:41:27.550803 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260620 23:41:27.551396 23199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:27.541621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55730 (local address 127.0.97.62:46097)
0620 23:41:27.543227 (+ 1606us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:27.543233 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:27.543251 (+ 18us) server_negotiation.cc:407] Connection header received
0620 23:41:27.543370 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:27.543378 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:27.543621 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:27.543810 (+ 189us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:41:27.545079 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:27.546261 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:27.548376 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:27.548985 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:27.549104 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:27.549936 (+ 832us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:41:27.550383 (+ 447us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:41:27.550476 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 23:41:27.551015 (+ 539us) server_negotiation.cc:299] Negotiation successful
0620 23:41:27.551115 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1485,"thread_start_us":1789,"threads_started":1}
I20260620 23:41:27.555781 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 22806
I20260620 23:41:27.557335 22868 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:41:27.671849 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 22806
I20260620 23:41:27.694278 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 22880
I20260620 23:41:27.695549 22942 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:41:27.806797 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 22880
I20260620 23:41:27.827438 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 22955
I20260620 23:41:27.828630 23017 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:41:27.939551 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 22955
2026-06-20T23:41:27Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4614 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:41:28 dist-test-slave-tcgx krb5kdc[23207](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:41:28 dist-test-slave-tcgx krb5kdc[23207](info): set up 2 sockets
Jun 20 23:41:28 dist-test-slave-tcgx krb5kdc[23207](info): commencing operation
krb5kdc: starting...
W20260620 23:41:29.063699 388 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/krb5kdc/test-user.keytab.
Jun 20 23:41:29 dist-test-slave-tcgx krb5kdc[23207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:41:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:41:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 23:41:29.338896 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39387
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39387,127.0.97.61:39781,127.0.97.60:45181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:41:29.686877 23220 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 23:41:29.687304 23220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:41:29.687435 23220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:41:29.687570 23220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:41:29.695094 23220 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:41:29.695200 23220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:41:29.695302 23220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:41:29.695392 23220 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:41:29.695510 23220 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:41:29.705780 23220 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39387,127.0.97.61:39781,127.0.97.60:45181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:41:29.707262 23220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:41:29.708674 23220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:41:29.721539 23228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:41:29.721632 23225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:41:29.721632 23226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:41:29.723058 23220 server_base.cc:1034] running on GCE node
I20260620 23:41:29.723955 23220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:41:29.725427 23220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:41:29.726680 23220 hybrid_clock.cc:648] HybridClock initialized: now 1781998889726618 us; error 37 us; skew 500 ppm
I20260620 23:41:29.727075 23220 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:41:29 dist-test-slave-tcgx krb5kdc[23207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998889, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:41:29.730325 23220 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 23:41:29.732074 23220 webserver.cc:460] Webserver started at http://127.0.97.62:44065/ using document root <none> and password file <none>
I20260620 23:41:29.732851 23220 fs_manager.cc:359] Metadata directory not provided
I20260620 23:41:29.732995 23220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:41:29.733328 23220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:41:29.736559 23220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/minicluster-data/master-0/data/instance:
uuid: "2431d6296f234c0681b9a99f8064d195"
format_stamp: "Formatted at 2026-06-20 23:41:29 on dist-test-slave-tcgx"
server_key: "cee27ea3a4aace436ffd19dee2faffe9"
server_key_iv: "d69c92b1e33f175fadd2cfe32284ee78"
server_key_version: "encryptionkey@0"
I20260620 23:41:29.737414 23220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/minicluster-data/master-0/wal/instance:
uuid: "2431d6296f234c0681b9a99f8064d195"
format_stamp: "Formatted at 2026-06-20 23:41:29 on dist-test-slave-tcgx"
server_key: "cee27ea3a4aace436ffd19dee2faffe9"
server_key_iv: "d69c92b1e33f175fadd2cfe32284ee78"
server_key_version: "encryptionkey@0"
I20260620 23:41:29.743120 23220 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 23:41:29.747062 23235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:41:29.748672 23220 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 23:41:29.748906 23220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/minicluster-data/master-0/wal
uuid: "2431d6296f234c0681b9a99f8064d195"
format_stamp: "Formatted at 2026-06-20 23:41:29 on dist-test-slave-tcgx"
server_key: "cee27ea3a4aace436ffd19dee2faffe9"
server_key_iv: "d69c92b1e33f175fadd2cfe32284ee78"
server_key_version: "encryptionkey@0"
I20260620 23:41:29.749157 23220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:41:29.766850 23220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:41:29.771607 23220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:41:29.771917 23220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:41:29.791520 23220 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39387
I20260620 23:41:29.791544 23286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39387 every 8 connection(s)
I20260620 23:41:29.793053 23220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/minicluster-data/master-0/data/info.pb
I20260620 23:41:29.795815 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 23220
I20260620 23:41:29.796012 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/minicluster-data/master-0/wal/instance
I20260620 23:41:29.796540 388 external_mini_cluster.cc:1428] Setting key e4c854898e80e46945d733f4c8d0d5c3
I20260620 23:41:29.797628 23287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:41:29.808115 23287 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39387 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:41:29 dist-test-slave-tcgx krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998889, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260620 23:41:29.833163 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:29.811725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36820 (local address 127.0.97.62:39387)
0620 23:41:29.812762 (+ 1037us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:29.812782 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:29.813293 (+ 511us) server_negotiation.cc:407] Connection header received
0620 23:41:29.815552 (+ 2259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:29.815559 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:29.815846 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:29.816063 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:29.817451 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:29.818763 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:29.820288 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:29.821384 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:29.821666 (+ 282us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:29.825265 (+ 3599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:41:29.825274 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:41:29.825363 (+ 89us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:41:29.825452 (+ 89us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:41:29.828499 (+ 3047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:29.829526 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:29.829533 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:29.829535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:29.829704 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:29.830226 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:29.830231 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:29.830233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:29.830802 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:41:29.831059 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0620 23:41:29.831722 (+ 663us) server_negotiation.cc:299] Negotiation successful
0620 23:41:29.832019 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":744,"thread_start_us":511,"threads_started":1}
I20260620 23:41:29.833230 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:29.811725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39387 (local address 127.0.0.1:36820)
0620 23:41:29.813123 (+ 1398us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:29.813160 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 23:41:29.815129 (+ 1969us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:41:29.816079 (+ 950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:41:29.816085 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:41:29.816866 (+ 781us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:41:29.817220 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:41:29.817239 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:41:29.818924 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:41:29.818955 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:41:29.820077 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:41:29.820090 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:41:29.821653 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:41:29.821659 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:41:29.821870 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:29.822598 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:41:29.822625 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:41:29.825001 (+ 2376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:41:29.828675 (+ 3674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:41:29.828690 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:41:29.828709 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:41:29.829342 (+ 633us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:41:29.829827 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:41:29.829834 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:41:29.829840 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:41:29.830056 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:41:29.831002 (+ 946us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:41:29.831014 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:41:29.831295 (+ 281us) client_negotiation.cc:769] Sending connection context
0620 23:41:29.831579 (+ 284us) client_negotiation.cc:240] Negotiation successful
0620 23:41:29.832226 (+ 647us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1172,"mutex_wait_us":133,"thread_start_us":960,"threads_started":1}
I20260620 23:41:29.836809 23287 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39781 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:41:29.838047 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:29.837617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39781 (local address 127.0.0.1:50068)
0620 23:41:29.837796 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:29.837919 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260620 23:41:29.838376 23238 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39781: connect: Connection refused (error 111)
I20260620 23:41:29.840080 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:29.839712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39781 (local address 127.0.0.1:50080)
0620 23:41:29.839856 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:29.839929 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 23:41:29.840409 23287 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:39781: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39781: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260620 23:41:29.863026 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:39781
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39387,127.0.97.61:39781,127.0.97.60:45181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:41:29.889825 23287 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39781 } attempt: 1
I20260620 23:41:29.892421 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:29.890638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39781 (local address 127.0.0.1:50084)
0620 23:41:29.892066 (+ 1428us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:29.892237 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1260}
I20260620 23:41:29.893544 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:29.893166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39781 (local address 127.0.0.1:50094)
0620 23:41:29.893322 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:29.893406 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 23:41:29.893728 23287 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:39781: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39781: connect: Connection refused (error 111)
I20260620 23:41:29.961990 23287 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39781 } attempt: 2
I20260620 23:41:29.963378 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:29.962789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39781 (local address 127.0.0.1:50108)
0620 23:41:29.963080 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:29.963185 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 23:41:29.964485 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:29.964126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39781 (local address 127.0.0.1:50124)
0620 23:41:29.964277 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:29.964347 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 23:41:29.964715 23287 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:39781: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39781: connect: Connection refused (error 111)
I20260620 23:41:30.055996 23287 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39781 } attempt: 3
I20260620 23:41:30.057294 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:30.056812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39781 (local address 127.0.0.1:50128)
0620 23:41:30.057037 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:30.057141 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260620 23:41:30.058625 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:30.058163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39781 (local address 127.0.0.1:50136)
0620 23:41:30.058332 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:30.058474 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260620 23:41:30.058926 23287 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:39781: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39781: connect: Connection refused (error 111)
I20260620 23:41:30.202258 23287 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39781 } attempt: 4
I20260620 23:41:30.203874 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:30.203080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39781 (local address 127.0.0.1:50142)
0620 23:41:30.203510 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:30.203648 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232}
I20260620 23:41:30.205034 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:30.204646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39781 (local address 127.0.0.1:50146)
0620 23:41:30.204801 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:30.204891 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 23:41:30.205376 23287 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:39781: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39781: connect: Connection refused (error 111)
W20260620 23:41:30.211675 23294 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 23:41:30.212087 23294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:41:30.212212 23294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:41:30.212352 23294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:41:30.220141 23294 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:41:30.220311 23294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:41:30.220432 23294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:41:30.220518 23294 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:41:30.220616 23294 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:41:30.230952 23294 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39387,127.0.97.61:39781,127.0.97.60:45181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:39781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:41:30.232419 23294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:41:30.233853 23294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:41:30.246572 23302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:41:30.246654 23299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:41:30.246891 23300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:41:30.247771 23294 server_base.cc:1034] running on GCE node
I20260620 23:41:30.248633 23294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:41:30.250312 23294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:41:30.251590 23294 hybrid_clock.cc:648] HybridClock initialized: now 1781998890251509 us; error 44 us; skew 500 ppm
I20260620 23:41:30.251950 23294 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:41:30 dist-test-slave-tcgx krb5kdc[23207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998890, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:41:30.255478 23294 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260620 23:41:30.257133 23294 webserver.cc:460] Webserver started at http://127.0.97.61:34117/ using document root <none> and password file <none>
I20260620 23:41:30.257920 23294 fs_manager.cc:359] Metadata directory not provided
I20260620 23:41:30.258064 23294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:41:30.258401 23294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:41:30.261675 23294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/minicluster-data/master-1/data/instance:
uuid: "0a5d1da7ecb34d4688d0fda3b6752345"
format_stamp: "Formatted at 2026-06-20 23:41:30 on dist-test-slave-tcgx"
server_key: "7377b6949c9afcd81597e89dd28170d2"
server_key_iv: "11c8e6940da171fd13bfc9aefa6f73b7"
server_key_version: "encryptionkey@0"
I20260620 23:41:30.262496 23294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/minicluster-data/master-1/wal/instance:
uuid: "0a5d1da7ecb34d4688d0fda3b6752345"
format_stamp: "Formatted at 2026-06-20 23:41:30 on dist-test-slave-tcgx"
server_key: "7377b6949c9afcd81597e89dd28170d2"
server_key_iv: "11c8e6940da171fd13bfc9aefa6f73b7"
server_key_version: "encryptionkey@0"
I20260620 23:41:30.268457 23294 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260620 23:41:30.272354 23309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:41:30.273841 23294 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:41:30.274047 23294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/minicluster-data/master-1/wal
uuid: "0a5d1da7ecb34d4688d0fda3b6752345"
format_stamp: "Formatted at 2026-06-20 23:41:30 on dist-test-slave-tcgx"
server_key: "7377b6949c9afcd81597e89dd28170d2"
server_key_iv: "11c8e6940da171fd13bfc9aefa6f73b7"
server_key_version: "encryptionkey@0"
I20260620 23:41:30.274293 23294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:41:30.297753 23294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:41:30.302130 23294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:41:30.302430 23294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:41:30.322189 23294 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:39781
I20260620 23:41:30.322222 23360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:39781 every 8 connection(s)
I20260620 23:41:30.323632 23294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/minicluster-data/master-1/data/info.pb
I20260620 23:41:30.327924 23361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:41:30.330371 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 23294
I20260620 23:41:30.330574 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/minicluster-data/master-1/wal/instance
I20260620 23:41:30.331154 388 external_mini_cluster.cc:1428] Setting key 595d9cbeb6b0d6f23fbdc2b7f8ab5af8
I20260620 23:41:30.337670 23361 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39387 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:41:30 dist-test-slave-tcgx krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998890, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260620 23:41:30.360795 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:30.341386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36824 (local address 127.0.97.62:39387)
0620 23:41:30.341984 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:30.341989 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:30.343174 (+ 1185us) server_negotiation.cc:407] Connection header received
0620 23:41:30.345169 (+ 1995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:30.345176 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:30.345283 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:30.345410 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:30.347077 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:30.347888 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:30.349497 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:30.349925 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:30.350069 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:30.354081 (+ 4012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:41:30.354086 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:41:30.354089 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:41:30.354130 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:41:30.357158 (+ 3028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:30.358128 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:30.358133 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:30.358136 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:30.358266 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:30.358743 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:30.358752 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:30.358754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:30.359126 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:41:30.359298 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 23:41:30.360123 (+ 825us) server_negotiation.cc:299] Negotiation successful
0620 23:41:30.360370 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":278,"threads_started":1}
I20260620 23:41:30.361802 23365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:30.342074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39387 (local address 127.0.0.1:36824)
0620 23:41:30.342994 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:30.343053 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 23:41:30.344863 (+ 1810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:41:30.345665 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:41:30.345674 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:41:30.346245 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:41:30.346833 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:41:30.346852 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:41:30.348077 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:41:30.348081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:41:30.349338 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:41:30.349349 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:41:30.350171 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:41:30.350178 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:41:30.350423 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:30.351214 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:41:30.351237 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:41:30.353813 (+ 2576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:41:30.357387 (+ 3574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:41:30.357398 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:41:30.357420 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:41:30.357971 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:41:30.358399 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:41:30.358405 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:41:30.358409 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:41:30.358611 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:41:30.359287 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:41:30.359295 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:41:30.359701 (+ 406us) client_negotiation.cc:769] Sending connection context
0620 23:41:30.360021 (+ 320us) client_negotiation.cc:240] Negotiation successful
0620 23:41:30.360627 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":384,"threads_started":1}
I20260620 23:41:30.363904 23361 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39781 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:41:30 dist-test-slave-tcgx krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998890, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 23:41:30.381831 23366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:30.364865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50150 (local address 127.0.97.61:39781)
0620 23:41:30.367422 (+ 2557us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:30.367429 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:30.367465 (+ 36us) server_negotiation.cc:407] Connection header received
0620 23:41:30.367538 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:30.367544 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:30.367754 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:30.367950 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:30.368737 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:30.369812 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:30.370989 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:30.371592 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:30.371736 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:30.375312 (+ 3576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:41:30.375326 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:41:30.375341 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:41:30.375371 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:41:30.378595 (+ 3224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:30.379577 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:30.379584 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:30.379586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:30.379706 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:30.380153 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:30.380156 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:30.380158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:30.380536 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:41:30.380756 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 23:41:30.381171 (+ 415us) server_negotiation.cc:299] Negotiation successful
0620 23:41:30.381525 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":197,"server-negotiator.queue_time_us":2402,"thread_start_us":2264,"threads_started":1}
I20260620 23:41:30.381951 23365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:30.364803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39781 (local address 127.0.0.1:50150)
0620 23:41:30.364997 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:30.365017 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:41:30.365121 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:41:30.367913 (+ 2792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:41:30.367918 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:41:30.368410 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:41:30.368554 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:41:30.368563 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:41:30.369967 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:41:30.369974 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:41:30.370813 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:41:30.370824 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:41:30.371746 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:41:30.371752 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:41:30.371963 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:30.372617 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:41:30.372635 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:41:30.375066 (+ 2431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:41:30.378778 (+ 3712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:41:30.378786 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:41:30.378793 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:41:30.379418 (+ 625us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:41:30.379826 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:41:30.379833 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:41:30.379840 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:41:30.380045 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:41:30.380675 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:41:30.380683 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:41:30.380879 (+ 196us) client_negotiation.cc:769] Sending connection context
0620 23:41:30.381043 (+ 164us) client_negotiation.cc:240] Negotiation successful
0620 23:41:30.381409 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":3}
I20260620 23:41:30.384717 23361 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45181 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:41:30.386188 23365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:30.385690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45181 (local address 127.0.0.1:60198)
0620 23:41:30.385905 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:30.386030 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260620 23:41:30.386492 23313 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45181: connect: Connection refused (error 111)
I20260620 23:41:30.388154 23365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:30.387805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45181 (local address 127.0.0.1:60200)
0620 23:41:30.387971 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:30.388050 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 23:41:30.388366 23361 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45181: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45181: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260620 23:41:30.398335 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
/tmp/dist-test-taskN6eSV7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:45181
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39387,127.0.97.61:39781,127.0.97.60:45181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:41:30.437777 23361 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45181 } attempt: 1
I20260620 23:41:30.438964 23365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:30.438456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45181 (local address 127.0.0.1:60208)
0620 23:41:30.438667 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:30.438776 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 23:41:30.440354 23365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:30.439840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45181 (local address 127.0.0.1:60212)
0620 23:41:30.440079 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:30.440191 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260620 23:41:30.440631 23361 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45181: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45181: connect: Connection refused (error 111)
I20260620 23:41:30.504801 23287 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39781 } attempt: 5
I20260620 23:41:30.508869 23361 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45181 } attempt: 2
I20260620 23:41:30.509895 23365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:30.509451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45181 (local address 127.0.0.1:60224)
0620 23:41:30.509642 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:30.509736 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 23:41:30.511086 23365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:30.510632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45181 (local address 127.0.0.1:60230)
0620 23:41:30.510802 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:30.510901 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260620 23:41:30.511672 23361 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45181: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45181: connect: Connection refused (error 111)
Jun 20 23:41:30 dist-test-slave-tcgx krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998889, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 23:41:30.519972 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:30.505543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39781 (local address 127.0.0.1:50158)
0620 23:41:30.505787 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:30.505811 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 23:41:30.505959 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:41:30.506394 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:41:30.506400 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:41:30.506990 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:41:30.507230 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:41:30.507242 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:41:30.508324 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:41:30.508332 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:41:30.509830 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:41:30.509859 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:41:30.510990 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:41:30.510996 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:41:30.511277 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:30.511938 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:41:30.511956 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:41:30.514038 (+ 2082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:41:30.517339 (+ 3301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:41:30.517345 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:41:30.517349 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:41:30.517757 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:41:30.518141 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:41:30.518146 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:41:30.518150 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:41:30.518375 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:41:30.519170 (+ 795us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:41:30.519174 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:41:30.519311 (+ 137us) client_negotiation.cc:769] Sending connection context
0620 23:41:30.519455 (+ 144us) client_negotiation.cc:240] Negotiation successful
0620 23:41:30.519721 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260620 23:41:30.520349 23366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:30.505803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50158 (local address 127.0.97.61:39781)
0620 23:41:30.505986 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:30.505991 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:30.506013 (+ 22us) server_negotiation.cc:407] Connection header received
0620 23:41:30.506121 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:30.506128 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:30.506215 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:30.506309 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:30.507400 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:30.508115 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:30.510099 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:30.510815 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:30.510973 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:30.514274 (+ 3301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:41:30.514280 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:41:30.514283 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:41:30.514318 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:41:30.517196 (+ 2878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:30.517882 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:30.517889 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:30.517892 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:30.518021 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:30.518569 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:30.518575 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:30.518578 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:30.519037 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:41:30.519195 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 23:41:30.519628 (+ 433us) server_negotiation.cc:299] Negotiation successful
0620 23:41:30.519874 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 23:41:30.521421 23287 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45181 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:41:30.522449 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:30.522057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45181 (local address 127.0.0.1:60232)
0620 23:41:30.522233 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:30.522317 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 23:41:30.523397 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:30.523060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45181 (local address 127.0.0.1:60246)
0620 23:41:30.523218 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:30.523304 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260620 23:41:30.523665 23287 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45181: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45181: connect: Connection refused (error 111)
I20260620 23:41:30.574973 23287 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45181 } attempt: 1
I20260620 23:41:30.576189 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:30.575730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45181 (local address 127.0.0.1:60262)
0620 23:41:30.575932 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:30.576023 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260620 23:41:30.577210 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:30.576923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45181 (local address 127.0.0.1:60272)
0620 23:41:30.577050 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:30.577116 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 23:41:30.577455 23287 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45181: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45181: connect: Connection refused (error 111)
I20260620 23:41:30.602977 23361 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45181 } attempt: 3
I20260620 23:41:30.604473 23365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:30.603789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45181 (local address 127.0.0.1:60286)
0620 23:41:30.604081 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:30.604226 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260620 23:41:30.605767 23365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:30.605259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45181 (local address 127.0.0.1:60292)
0620 23:41:30.605516 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:30.605611 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
W20260620 23:41:30.605984 23361 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45181: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45181: connect: Connection refused (error 111)
I20260620 23:41:30.645689 23287 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45181 } attempt: 2
I20260620 23:41:30.647078 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:30.646506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45181 (local address 127.0.0.1:60306)
0620 23:41:30.646760 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:30.646850 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260620 23:41:30.648092 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:30.647774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45181 (local address 127.0.0.1:60310)
0620 23:41:30.647914 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:30.647991 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 23:41:30.648293 23287 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45181: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45181: connect: Connection refused (error 111)
I20260620 23:41:30.749246 23361 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45181 } attempt: 4
I20260620 23:41:30.750636 23365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:30.750028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45181 (local address 127.0.0.1:60322)
0620 23:41:30.750301 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:30.750419 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260620 23:41:30.751899 23365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:30.751515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45181 (local address 127.0.0.1:60328)
0620 23:41:30.751676 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:30.751761 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 23:41:30.752206 23361 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45181: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45181: connect: Connection refused (error 111)
I20260620 23:41:30.754485 23287 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45181 } attempt: 3
I20260620 23:41:30.755646 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:30.755139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45181 (local address 127.0.0.1:60330)
0620 23:41:30.755340 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:30.755440 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 23:41:30.756735 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:30.756297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45181 (local address 127.0.0.1:60340)
0620 23:41:30.756516 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:30.756606 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260620 23:41:30.756945 23287 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45181: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45181: connect: Connection refused (error 111)
W20260620 23:41:30.758688 23369 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 23:41:30.759090 23369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:41:30.759197 23369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:41:30.759332 23369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:41:30.767321 23369 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:41:30.767485 23369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:41:30.767592 23369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:41:30.767690 23369 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:41:30.767781 23369 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:41:30.778312 23369 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39387,127.0.97.61:39781,127.0.97.60:45181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:45181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:41:30.779838 23369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:41:30.781257 23369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:41:30.793756 23377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:41:30.794124 23374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:41:30.793805 23375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:41:30.794924 23369 server_base.cc:1034] running on GCE node
I20260620 23:41:30.795805 23369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:41:30.797401 23369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:41:30.798666 23369 hybrid_clock.cc:648] HybridClock initialized: now 1781998890798584 us; error 58 us; skew 500 ppm
I20260620 23:41:30.799081 23369 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:41:30 dist-test-slave-tcgx krb5kdc[23207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998890, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:41:30.806491 23369 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260620 23:41:30.808157 23369 webserver.cc:460] Webserver started at http://127.0.97.60:40191/ using document root <none> and password file <none>
I20260620 23:41:30.808941 23369 fs_manager.cc:359] Metadata directory not provided
I20260620 23:41:30.809078 23369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:41:30.809415 23369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:41:30.812628 23369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/minicluster-data/master-2/data/instance:
uuid: "df9835b6a1eb410a80c2f0fe4201ee3e"
format_stamp: "Formatted at 2026-06-20 23:41:30 on dist-test-slave-tcgx"
server_key: "2a04d2a8a2874290b6f121491749278a"
server_key_iv: "c51eea92effca1bd6e339fecf6cb60e7"
server_key_version: "encryptionkey@0"
I20260620 23:41:30.813470 23369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/minicluster-data/master-2/wal/instance:
uuid: "df9835b6a1eb410a80c2f0fe4201ee3e"
format_stamp: "Formatted at 2026-06-20 23:41:30 on dist-test-slave-tcgx"
server_key: "2a04d2a8a2874290b6f121491749278a"
server_key_iv: "c51eea92effca1bd6e339fecf6cb60e7"
server_key_version: "encryptionkey@0"
I20260620 23:41:30.818907 23369 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 23:41:30.822692 23384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:41:30.824210 23369 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 23:41:30.824412 23369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/minicluster-data/master-2/wal
uuid: "df9835b6a1eb410a80c2f0fe4201ee3e"
format_stamp: "Formatted at 2026-06-20 23:41:30 on dist-test-slave-tcgx"
server_key: "2a04d2a8a2874290b6f121491749278a"
server_key_iv: "c51eea92effca1bd6e339fecf6cb60e7"
server_key_version: "encryptionkey@0"
I20260620 23:41:30.824642 23369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:41:30.866999 23369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:41:30.871377 23369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:41:30.871678 23369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:41:30.890641 23369 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:45181
I20260620 23:41:30.890678 23435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:45181 every 8 connection(s)
I20260620 23:41:30.892167 23369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/minicluster-data/master-2/data/info.pb
I20260620 23:41:30.896555 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu as pid 23369
I20260620 23:41:30.896762 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskN6eSV7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998525501163-388-0/minicluster-data/master-2/wal/instance
I20260620 23:41:30.897179 23436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:41:30.897218 388 external_mini_cluster.cc:1428] Setting key 002ef88288ad68ba9cdb0b633d630da0
I20260620 23:41:30.906831 23436 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39387 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:41:30 dist-test-slave-tcgx krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 20 23:41:30 dist-test-slave-tcgx krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998890, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:41:30.921360 23439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:30.899122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60350 (local address 127.0.97.60:45181)
0620 23:41:30.899983 (+ 861us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:30.899996 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:30.900039 (+ 43us) server_negotiation.cc:407] Connection header received
0620 23:41:30.902040 (+ 2001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:30.902064 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:30.902496 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:30.902863 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:30.904719 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:30.906004 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:30.907892 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:30.908667 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:30.908804 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:30.913606 (+ 4802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:41:30.913614 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:41:30.913626 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:41:30.913666 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:41:30.916875 (+ 3209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:30.917630 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:30.917637 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:30.917647 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:30.917810 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:30.918220 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:30.918224 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:30.918226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:30.918851 (+ 625us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:41:30.919201 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0620 23:41:30.919665 (+ 464us) server_negotiation.cc:299] Negotiation successful
0620 23:41:30.920118 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":411,"threads_started":1}
I20260620 23:41:30.925467 23441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:30.910066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39387 (local address 127.0.0.1:36836)
0620 23:41:30.911055 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:30.911104 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 23:41:30.911268 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:41:30.911709 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:41:30.911714 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:41:30.912309 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:41:30.912640 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:41:30.912651 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:41:30.913891 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:41:30.913897 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:41:30.915432 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:41:30.915449 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:41:30.916240 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:41:30.916247 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:41:30.916439 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:30.917055 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:41:30.917072 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:41:30.919315 (+ 2243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:41:30.922491 (+ 3176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:41:30.922504 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:41:30.922512 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:41:30.923041 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:41:30.923432 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:41:30.923437 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:41:30.923440 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:41:30.923575 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:41:30.924227 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:41:30.924237 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:41:30.924457 (+ 220us) client_negotiation.cc:769] Sending connection context
0620 23:41:30.924664 (+ 207us) client_negotiation.cc:240] Negotiation successful
0620 23:41:30.925046 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"thread_start_us":552,"threads_started":1}
I20260620 23:41:30.926390 23440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:30.910072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36836 (local address 127.0.97.62:39387)
0620 23:41:30.910586 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:30.910594 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:30.911203 (+ 609us) server_negotiation.cc:407] Connection header received
0620 23:41:30.911398 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:30.911404 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:30.911505 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:30.911625 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:30.912801 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:30.913728 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:30.915619 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:30.916034 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:30.916190 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:30.919514 (+ 3324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:41:30.919519 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:41:30.919522 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:41:30.919565 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:41:30.922350 (+ 2785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:30.923185 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:30.923189 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:30.923192 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:30.923315 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:30.923698 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:30.923708 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:30.923710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:30.924056 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:41:30.924221 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 23:41:30.925831 (+ 1610us) server_negotiation.cc:299] Negotiation successful
0620 23:41:30.926120 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":376,"thread_start_us":257,"threads_started":1}
I20260620 23:41:30.927863 23436 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39781 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:41:30 dist-test-slave-tcgx krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 23:41:30.934158 23287 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45181 } attempt: 4
Jun 20 23:41:30 dist-test-slave-tcgx krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998890, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Jun 20 23:41:30 dist-test-slave-tcgx krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998889, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260620 23:41:30.943156 23442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:30.924508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36840 (local address 127.0.97.62:39387)
0620 23:41:30.925158 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:30.925165 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:30.925189 (+ 24us) server_negotiation.cc:407] Connection header received
0620 23:41:30.925239 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:30.925244 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:30.925314 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:30.925425 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:30.926753 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:30.927802 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:30.929503 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:30.929992 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:30.930084 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:30.934774 (+ 4690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:41:30.934784 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:41:30.934789 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:41:30.934832 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:41:30.939868 (+ 5036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:30.940800 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:30.940805 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:30.940808 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:30.940979 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:30.941423 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:30.941428 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:30.941433 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:30.941806 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:41:30.942151 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0620 23:41:30.942403 (+ 252us) server_negotiation.cc:299] Negotiation successful
0620 23:41:30.942645 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":496,"thread_start_us":273,"threads_started":1}
I20260620 23:41:30.943588 23366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:30.928695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50172 (local address 127.0.97.61:39781)
0620 23:41:30.928882 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:30.928888 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:30.928907 (+ 19us) server_negotiation.cc:407] Connection header received
0620 23:41:30.929092 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:30.929096 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:30.929162 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:30.929243 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:30.930034 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:30.931073 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:30.932377 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:30.933088 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:30.933226 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:30.937000 (+ 3774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:41:30.937007 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:41:30.937011 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:41:30.937053 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:41:30.940458 (+ 3405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:30.941274 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:30.941280 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:30.941283 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:30.941464 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:30.941859 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:30.941864 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:30.941868 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:30.942209 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:41:30.942346 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 23:41:30.943047 (+ 701us) server_negotiation.cc:299] Negotiation successful
0620 23:41:30.943223 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 23:41:30.944563 23441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:30.928657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39781 (local address 127.0.0.1:50172)
0620 23:41:30.928840 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:30.928860 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:41:30.928954 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:41:30.929300 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:41:30.929306 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:41:30.929736 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:41:30.929878 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:41:30.929886 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:41:30.931264 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:41:30.931269 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:41:30.932206 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:41:30.932218 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:41:30.933257 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:41:30.933263 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:41:30.933464 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:30.934072 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:41:30.934116 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:41:30.936760 (+ 2644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:41:30.940638 (+ 3878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:41:30.940643 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:41:30.940646 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:41:30.941123 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:41:30.941601 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:41:30.941604 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:41:30.941606 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:41:30.941737 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:41:30.942408 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:41:30.942413 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:41:30.942568 (+ 155us) client_negotiation.cc:769] Sending connection context
0620 23:41:30.943844 (+ 1276us) client_negotiation.cc:240] Negotiation successful
0620 23:41:30.944190 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 23:41:30.945688 23436 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45181 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:41:30.951382 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:30.934795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45181 (local address 127.0.0.1:60360)
0620 23:41:30.935025 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:30.935049 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 23:41:30.935166 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:41:30.935992 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:41:30.935999 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:41:30.936470 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:41:30.936712 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:41:30.936725 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:41:30.938070 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:41:30.938074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:41:30.939308 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:41:30.939321 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:41:30.940021 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:41:30.940026 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:41:30.940220 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:30.940766 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:41:30.940792 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:41:30.943817 (+ 3025us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:41:30.948484 (+ 4667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:41:30.948489 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:41:30.948493 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:41:30.949045 (+ 552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:41:30.949464 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:41:30.949469 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:41:30.949474 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:41:30.949669 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:41:30.950325 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:41:30.950330 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:41:30.950500 (+ 170us) client_negotiation.cc:769] Sending connection context
0620 23:41:30.950652 (+ 152us) client_negotiation.cc:240] Negotiation successful
0620 23:41:30.950952 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260620 23:41:30.951370 23439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:30.935206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60360 (local address 127.0.97.60:45181)
0620 23:41:30.935374 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:30.935378 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:30.935392 (+ 14us) server_negotiation.cc:407] Connection header received
0620 23:41:30.935428 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:30.935431 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:30.935486 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:30.935943 (+ 457us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:30.936859 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:30.937917 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:30.939461 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:30.939866 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:30.940008 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:30.943996 (+ 3988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:41:30.944002 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:41:30.944006 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:41:30.944045 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:41:30.948344 (+ 4299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:30.949192 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:30.949197 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:30.949200 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:30.949334 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:30.949808 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:30.949813 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:30.949816 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:30.950200 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:41:30.950360 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 23:41:30.950760 (+ 400us) server_negotiation.cc:299] Negotiation successful
0620 23:41:30.951020 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
Jun 20 23:41:30 dist-test-slave-tcgx krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Jun 20 23:41:30 dist-test-slave-tcgx krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998890, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260620 23:41:30.957466 23287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2431d6296f234c0681b9a99f8064d195: Bootstrap starting.
I20260620 23:41:30.961218 23287 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2431d6296f234c0681b9a99f8064d195: Neither blocks nor log segments found. Creating new log.
I20260620 23:41:30.962663 23287 log.cc:824] T 00000000000000000000000000000000 P 2431d6296f234c0681b9a99f8064d195: Log is configured to *not* fsync() on all Append() calls
I20260620 23:41:30.963761 23441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:30.946432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45181 (local address 127.0.0.1:60364)
0620 23:41:30.946592 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:30.946618 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 23:41:30.946719 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:41:30.947757 (+ 1038us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:41:30.947763 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:41:30.948362 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:41:30.948546 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:41:30.948558 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:41:30.950172 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:41:30.950178 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:41:30.951655 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:41:30.951673 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:41:30.952682 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:41:30.952688 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:41:30.952914 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:30.953502 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:41:30.953525 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:41:30.956219 (+ 2694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:41:30.961008 (+ 4789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:41:30.961016 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:41:30.961023 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:41:30.961527 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:41:30.961931 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:41:30.961937 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:41:30.961940 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:41:30.962076 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:41:30.962690 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:41:30.962695 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:41:30.962843 (+ 148us) client_negotiation.cc:769] Sending connection context
0620 23:41:30.963062 (+ 219us) client_negotiation.cc:240] Negotiation successful
0620 23:41:30.963392 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 23:41:30.964519 23366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:30.945211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50182 (local address 127.0.97.61:39781)
0620 23:41:30.945372 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:30.945375 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:30.945583 (+ 208us) server_negotiation.cc:407] Connection header received
0620 23:41:30.945770 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:30.945775 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:30.945863 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:30.945957 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:30.947673 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:30.948775 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:30.949978 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:30.950632 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:30.950761 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:30.957582 (+ 6821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:41:30.957588 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:41:30.957592 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:41:30.957653 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:41:30.961025 (+ 3372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:30.961872 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:30.961878 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:30.961881 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:30.962050 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:30.962503 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:30.962508 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:30.962511 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:30.962913 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:41:30.963263 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0620 23:41:30.963582 (+ 319us) server_negotiation.cc:299] Negotiation successful
0620 23:41:30.963907 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 23:41:30.964553 23443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:30.946687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60364 (local address 127.0.97.60:45181)
0620 23:41:30.947473 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:30.947479 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:30.947499 (+ 20us) server_negotiation.cc:407] Connection header received
0620 23:41:30.947544 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:30.947549 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:30.947624 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:30.947729 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:30.948714 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:30.950045 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:30.951829 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:30.952515 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:30.952663 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:30.956448 (+ 3785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:41:30.956454 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:41:30.956459 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:41:30.956503 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:41:30.960834 (+ 4331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:30.961670 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:30.961675 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:30.961678 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:30.961835 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:30.962210 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:30.962214 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:30.962220 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:30.962582 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:41:30.962731 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 23:41:30.964078 (+ 1347us) server_negotiation.cc:299] Negotiation successful
0620 23:41:30.964283 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":421,"threads_started":1}
I20260620 23:41:30.971985 23436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df9835b6a1eb410a80c2f0fe4201ee3e: Bootstrap starting.
I20260620 23:41:30.972733 23287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2431d6296f234c0681b9a99f8064d195: No bootstrap required, opened a new log
I20260620 23:41:30.973155 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:41:30.976150 23436 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df9835b6a1eb410a80c2f0fe4201ee3e: Neither blocks nor log segments found. Creating new log.
I20260620 23:41:30.977579 23436 log.cc:824] T 00000000000000000000000000000000 P df9835b6a1eb410a80c2f0fe4201ee3e: Log is configured to *not* fsync() on all Append() calls
I20260620 23:41:30.975569 23262 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "2431d6296f234c0681b9a99f8064d195"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36840
I20260620 23:41:30.981447 23436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df9835b6a1eb410a80c2f0fe4201ee3e: No bootstrap required, opened a new log
I20260620 23:41:30.984040 23287 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2431d6296f234c0681b9a99f8064d195 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2431d6296f234c0681b9a99f8064d195" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39387 } } peers { permanent_uuid: "0a5d1da7ecb34d4688d0fda3b6752345" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39781 } } peers { permanent_uuid: "df9835b6a1eb410a80c2f0fe4201ee3e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45181 } }
I20260620 23:41:30.984498 23287 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2431d6296f234c0681b9a99f8064d195 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2431d6296f234c0681b9a99f8064d195, State: Initialized, Role: FOLLOWER
I20260620 23:41:30.985040 23287 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2431d6296f234c0681b9a99f8064d195 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2431d6296f234c0681b9a99f8064d195" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39387 } } peers { permanent_uuid: "0a5d1da7ecb34d4688d0fda3b6752345" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39781 } } peers { permanent_uuid: "df9835b6a1eb410a80c2f0fe4201ee3e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45181 } }
I20260620 23:41:30.986136 23446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2431d6296f234c0681b9a99f8064d195 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2431d6296f234c0681b9a99f8064d195" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39387 } } peers { permanent_uuid: "0a5d1da7ecb34d4688d0fda3b6752345" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39781 } } peers { permanent_uuid: "df9835b6a1eb410a80c2f0fe4201ee3e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45181 } } }
I20260620 23:41:30.986589 23446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2431d6296f234c0681b9a99f8064d195 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:41:30.987556 23287 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2431d6296f234c0681b9a99f8064d195 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:41:30.987663 23262 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2431d6296f234c0681b9a99f8064d195 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 23:41:30.987876 23262 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2431d6296f234c0681b9a99f8064d195 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:41:30.990877 23262 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2431d6296f234c0681b9a99f8064d195 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2431d6296f234c0681b9a99f8064d195" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39387 } } peers { permanent_uuid: "0a5d1da7ecb34d4688d0fda3b6752345" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39781 } } peers { permanent_uuid: "df9835b6a1eb410a80c2f0fe4201ee3e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45181 } }
I20260620 23:41:30.993491 23262 leader_election.cc:288] T 00000000000000000000000000000000 P 2431d6296f234c0681b9a99f8064d195 [CANDIDATE]: Term 1 election: Requested vote from peers 0a5d1da7ecb34d4688d0fda3b6752345 (127.0.97.61:39781), df9835b6a1eb410a80c2f0fe4201ee3e (127.0.97.60:45181)
I20260620 23:41:30.994102 23436 raft_consensus.cc:348] T 00000000000000000000000000000000 P df9835b6a1eb410a80c2f0fe4201ee3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2431d6296f234c0681b9a99f8064d195" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39387 } } peers { permanent_uuid: "0a5d1da7ecb34d4688d0fda3b6752345" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39781 } } peers { permanent_uuid: "df9835b6a1eb410a80c2f0fe4201ee3e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45181 } }
I20260620 23:41:30.994699 23436 raft_consensus.cc:729] T 00000000000000000000000000000000 P df9835b6a1eb410a80c2f0fe4201ee3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df9835b6a1eb410a80c2f0fe4201ee3e, State: Initialized, Role: FOLLOWER
I20260620 23:41:30.995386 23436 consensus_queue.cc:260] T 00000000000000000000000000000000 P df9835b6a1eb410a80c2f0fe4201ee3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2431d6296f234c0681b9a99f8064d195" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39387 } } peers { permanent_uuid: "0a5d1da7ecb34d4688d0fda3b6752345" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39781 } } peers { permanent_uuid: "df9835b6a1eb410a80c2f0fe4201ee3e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45181 } }
I20260620 23:41:30.999536 23451 sys_catalog.cc:455] T 00000000000000000000000000000000 P df9835b6a1eb410a80c2f0fe4201ee3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2431d6296f234c0681b9a99f8064d195" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39387 } } peers { permanent_uuid: "0a5d1da7ecb34d4688d0fda3b6752345" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39781 } } peers { permanent_uuid: "df9835b6a1eb410a80c2f0fe4201ee3e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45181 } } }
I20260620 23:41:31.000080 23451 sys_catalog.cc:458] T 00000000000000000000000000000000 P df9835b6a1eb410a80c2f0fe4201ee3e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:41:31.002485 23436 sys_catalog.cc:564] T 00000000000000000000000000000000 P df9835b6a1eb410a80c2f0fe4201ee3e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:41:31.002570 23411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2431d6296f234c0681b9a99f8064d195" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "df9835b6a1eb410a80c2f0fe4201ee3e"
I20260620 23:41:31.003360 23411 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df9835b6a1eb410a80c2f0fe4201ee3e [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:41:31.010432 23411 raft_consensus.cc:2454] T 00000000000000000000000000000000 P df9835b6a1eb410a80c2f0fe4201ee3e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2431d6296f234c0681b9a99f8064d195 in term 1.
I20260620 23:41:31.012107 23239 leader_election.cc:302] T 00000000000000000000000000000000 P 2431d6296f234c0681b9a99f8064d195 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2431d6296f234c0681b9a99f8064d195, df9835b6a1eb410a80c2f0fe4201ee3e; no voters:
I20260620 23:41:31.012913 23446 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2431d6296f234c0681b9a99f8064d195 [term 1 FOLLOWER]: Leader election won for term 1
W20260620 23:41:31.024942 23241 tablet.cc:2369] T 00000000000000000000000000000000 P 2431d6296f234c0681b9a99f8064d195: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:41:31.030779 23446 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2431d6296f234c0681b9a99f8064d195 [term 1 LEADER]: Becoming Leader. State: Replica: 2431d6296f234c0681b9a99f8064d195, State: Running, Role: LEADER
I20260620 23:41:31.032836 23446 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2431d6296f234c0681b9a99f8064d195 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2431d6296f234c0681b9a99f8064d195" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39387 } } peers { permanent_uuid: "0a5d1da7ecb34d4688d0fda3b6752345" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39781 } } peers { permanent_uuid: "df9835b6a1eb410a80c2f0fe4201ee3e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45181 } }
I20260620 23:41:31.051426 23361 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45181 } attempt: 5
I20260620 23:41:31.057097 23366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:31.018048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50186 (local address 127.0.97.61:39781)
0620 23:41:31.018312 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:31.018321 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:31.035888 (+ 17567us) server_negotiation.cc:407] Connection header received
0620 23:41:31.036235 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:31.036244 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:31.036341 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:31.036440 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:31.038961 (+ 2521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:31.040182 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:31.042634 (+ 2452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:31.043357 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:31.043476 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:31.047705 (+ 4229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:41:31.047713 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:41:31.047719 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:41:31.047773 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:41:31.053345 (+ 5572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:31.054305 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:31.054312 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:31.054316 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:31.054510 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:31.055058 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:31.055064 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:31.055068 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:31.055444 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:41:31.055700 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0620 23:41:31.056344 (+ 644us) server_negotiation.cc:299] Negotiation successful
0620 23:41:31.056667 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
Jun 20 23:41:31 dist-test-slave-tcgx krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998890, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260620 23:41:31.073129 23458 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2431d6296f234c0681b9a99f8064d195 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2431d6296f234c0681b9a99f8064d195. Latest consensus state: current_term: 1 leader_uuid: "2431d6296f234c0681b9a99f8064d195" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2431d6296f234c0681b9a99f8064d195" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39387 } } peers { permanent_uuid: "0a5d1da7ecb34d4688d0fda3b6752345" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39781 } } peers { permanent_uuid: "df9835b6a1eb410a80c2f0fe4201ee3e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45181 } } }
I20260620 23:41:31.073830 23442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:31.020966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36846 (local address 127.0.97.62:39387)
0620 23:41:31.021232 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:31.021246 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:31.044080 (+ 22834us) server_negotiation.cc:407] Connection header received
0620 23:41:31.044944 (+ 864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:31.044957 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:31.045080 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:31.045169 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:31.054708 (+ 9539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:31.055902 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:31.060332 (+ 4430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:31.061093 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:31.061250 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:31.065236 (+ 3986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:41:31.065245 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:41:31.065252 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:41:31.065301 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:41:31.069771 (+ 4470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:31.070767 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:31.070775 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:31.070779 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:31.071003 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:31.071516 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:31.071522 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:31.071526 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:31.071981 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:41:31.072281 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0620 23:41:31.072948 (+ 667us) server_negotiation.cc:299] Negotiation successful
0620 23:41:31.073301 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 23:41:31.073505 23471 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 23:41:31.074653 23458 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2431d6296f234c0681b9a99f8064d195 [sys.catalog]: This master's current role is: LEADER
I20260620 23:41:31.077042 23439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:31.052952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60370 (local address 127.0.97.60:45181)
0620 23:41:31.053137 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:31.053143 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:31.053165 (+ 22us) server_negotiation.cc:407] Connection header received
0620 23:41:31.053225 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:31.053230 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:31.053325 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:31.053427 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:31.054524 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:31.055644 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:31.057933 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:31.058645 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:31.058765 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:31.063628 (+ 4863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:41:31.063638 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:41:31.063646 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:41:31.063701 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:41:31.069039 (+ 5338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:31.070505 (+ 1466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:31.070514 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:31.070521 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:31.070712 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:31.074667 (+ 3955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:31.074675 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:31.074679 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:31.075234 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:41:31.075400 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 23:41:31.076359 (+ 959us) server_negotiation.cc:299] Negotiation successful
0620 23:41:31.076625 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 23:41:31.078708 23365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:31.052119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45181 (local address 127.0.0.1:60370)
0620 23:41:31.052561 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0620 23:41:31.052587 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 23:41:31.052713 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:41:31.053468 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:41:31.053473 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:41:31.054085 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:41:31.054367 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:41:31.054382 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:41:31.055779 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:41:31.055788 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:41:31.057776 (+ 1988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:41:31.057796 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:41:31.058780 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:41:31.058789 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:41:31.059113 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:31.059916 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:41:31.059948 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:41:31.063364 (+ 3416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:41:31.069499 (+ 6135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:41:31.069509 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:41:31.069516 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:41:31.070259 (+ 743us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:41:31.073941 (+ 3682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:41:31.073949 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:41:31.074221 (+ 272us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:41:31.074493 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:41:31.075526 (+ 1033us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:41:31.075547 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:41:31.075906 (+ 359us) client_negotiation.cc:769] Sending connection context
0620 23:41:31.077499 (+ 1593us) client_negotiation.cc:240] Negotiation successful
0620 23:41:31.078085 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277}
I20260620 23:41:31.085722 23361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a5d1da7ecb34d4688d0fda3b6752345: Bootstrap starting.
I20260620 23:41:31.090476 23361 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a5d1da7ecb34d4688d0fda3b6752345: Neither blocks nor log segments found. Creating new log.
I20260620 23:41:31.092973 23361 log.cc:824] T 00000000000000000000000000000000 P 0a5d1da7ecb34d4688d0fda3b6752345: Log is configured to *not* fsync() on all Append() calls
I20260620 23:41:31.081594 23220 master_runner.cc:186] Error getting master registration for 127.0.97.61:39781: OK, instance_id { permanent_uuid: "0a5d1da7ecb34d4688d0fda3b6752345" instance_seqno: 1781998890316386 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 23:41:31.100759 23443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:31.022030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60368 (local address 127.0.97.60:45181)
0620 23:41:31.022282 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:31.022292 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:31.037405 (+ 15113us) server_negotiation.cc:407] Connection header received
0620 23:41:31.038751 (+ 1346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:31.038759 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:31.038913 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:31.039019 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:31.053203 (+ 14184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:31.054511 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:31.060387 (+ 5876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:31.061168 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:31.061265 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:31.070887 (+ 9622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:41:31.070895 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:41:31.070900 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:41:31.070977 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:41:31.086195 (+ 15218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:31.093371 (+ 7176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:31.093380 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:31.093384 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:31.093603 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:41:31.094277 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:41:31.094287 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:41:31.094293 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:41:31.094952 (+ 659us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:41:31.095307 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0620 23:41:31.099775 (+ 4468us) server_negotiation.cc:299] Negotiation successful
0620 23:41:31.100205 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 23:41:31.105136 23220 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 23:41:31.106155 23472 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:41:31.115444 23361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a5d1da7ecb34d4688d0fda3b6752345: No bootstrap required, opened a new log
W20260620 23:41:31.120064 23481 catalog_manager.cc:1558] T 00000000000000000000000000000000 P df9835b6a1eb410a80c2f0fe4201ee3e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:41:31.120239 23481 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:41:31.122066 23472 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260620 23:41:31.123883 23390 tablet.cc:2369] T 00000000000000000000000000000000 P df9835b6a1eb410a80c2f0fe4201ee3e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:41:31.128157 23361 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a5d1da7ecb34d4688d0fda3b6752345 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2431d6296f234c0681b9a99f8064d195" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39387 } } peers { permanent_uuid: "0a5d1da7ecb34d4688d0fda3b6752345" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39781 } } peers { permanent_uuid: "df9835b6a1eb410a80c2f0fe4201ee3e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45181 } }
I20260620 23:41:31.129452 23361 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a5d1da7ecb34d4688d0fda3b6752345 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a5d1da7ecb34d4688d0fda3b6752345, State: Initialized, Role: FOLLOWER
I20260620 23:41:31.130117 23361 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a5d1da7ecb34d4688d0fda3b6752345 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2431d6296f234c0681b9a99f8064d195" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39387 } } peers { permanent_uuid: "0a5d1da7ecb34d4688d0fda3b6752345" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39781 } } peers { permanent_uuid: "df9835b6a1eb410a80c2f0fe4201ee3e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45181 } }
I20260620 23:41:31.137337 23483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a5d1da7ecb34d4688d0fda3b6752345 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2431d6296f234c0681b9a99f8064d195" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39387 } } peers { permanent_uuid: "0a5d1da7ecb34d4688d0fda3b6752345" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39781 } } peers { permanent_uuid: "df9835b6a1eb410a80c2f0fe4201ee3e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45181 } } }
I20260620 23:41:31.137871 23483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a5d1da7ecb34d4688d0fda3b6752345 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:41:31.140071 23361 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a5d1da7ecb34d4688d0fda3b6752345 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:41:31.140148 23336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2431d6296f234c0681b9a99f8064d195" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0a5d1da7ecb34d4688d0fda3b6752345"
I20260620 23:41:31.140703 23336 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a5d1da7ecb34d4688d0fda3b6752345 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:41:31.144496 23336 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0a5d1da7ecb34d4688d0fda3b6752345 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2431d6296f234c0681b9a99f8064d195 in term 1.
I20260620 23:41:31.126158 23369 master_runner.cc:186] Error getting master registration for 127.0.97.61:39781: OK, instance_id { permanent_uuid: "0a5d1da7ecb34d4688d0fda3b6752345" instance_seqno: 1781998890316386 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 23:41:31.161209 23369 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 23:41:31.166070 23411 raft_consensus.cc:1261] T 00000000000000000000000000000000 P df9835b6a1eb410a80c2f0fe4201ee3e [term 1 FOLLOWER]: Refusing update from remote peer 2431d6296f234c0681b9a99f8064d195: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:41:31.166602 23336 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0a5d1da7ecb34d4688d0fda3b6752345 [term 1 FOLLOWER]: Refusing update from remote peer 2431d6296f234c0681b9a99f8064d195: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:41:31.167584 23446 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2431d6296f234c0681b9a99f8064d195 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df9835b6a1eb410a80c2f0fe4201ee3e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:41:31.168113 23458 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2431d6296f234c0681b9a99f8064d195 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a5d1da7ecb34d4688d0fda3b6752345" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:41:31.197136 23451 sys_catalog.cc:455] T 00000000000000000000000000000000 P df9835b6a1eb410a80c2f0fe4201ee3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2431d6296f234c0681b9a99f8064d195. Latest consensus state: current_term: 1 leader_uuid: "2431d6296f234c0681b9a99f8064d195" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2431d6296f234c0681b9a99f8064d195" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39387 } } peers { permanent_uuid: "0a5d1da7ecb34d4688d0fda3b6752345" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39781 } } peers { permanent_uuid: "df9835b6a1eb410a80c2f0fe4201ee3e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45181 } } }
I20260620 23:41:31.197613 23451 sys_catalog.cc:458] T 00000000000000000000000000000000 P df9835b6a1eb410a80c2f0fe4201ee3e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:41:31.201639 23446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2431d6296f234c0681b9a99f8064d195 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2431d6296f234c0681b9a99f8064d195" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2431d6296f234c0681b9a99f8064d195" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39387 } } peers { permanent_uuid: "0a5d1da7ecb34d4688d0fda3b6752345" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39781 } } peers { permanent_uuid: "df9835b6a1eb410a80c2f0fe4201ee3e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45181 } } }
I20260620 23:41:31.202005 23446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2431d6296f234c0681b9a99f8064d195 [sys.catalog]: This master's current role is: LEADER
I20260620 23:41:31.203907 23483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a5d1da7ecb34d4688d0fda3b6752345 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2431d6296f234c0681b9a99f8064d195. Latest consensus state: current_term: 1 leader_uuid: "2431d6296f234c0681b9a99f8064d195" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2431d6296f234c0681b9a99f8064d195" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39387 } } peers { permanent_uuid: "0a5d1da7ecb34d4688d0fda3b6752345" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39781 } } peers { permanent_uuid: "df9835b6a1eb410a80c2f0fe4201ee3e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45181 } } }
I20260620 23:41:31.204247 23483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a5d1da7ecb34d4688d0fda3b6752345 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:41:31.208129 23490 mvcc.cc:205] Tried to move back new op lower bound from 7299067458184708096 to 7299067457704734720. Current Snapshot: MvccSnapshot[applied={T|T < 7299067458184708096}]
I20260620 23:41:31.208448 23458 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2431d6296f234c0681b9a99f8064d195 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2431d6296f234c0681b9a99f8064d195" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2431d6296f234c0681b9a99f8064d195" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39387 } } peers { permanent_uuid: "0a5d1da7ecb34d4688d0fda3b6752345" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39781 } } peers { permanent_uuid: "df9835b6a1eb410a80c2f0fe4201ee3e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45181 } } }
I20260620 23:41:31.208753 23458 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2431d6296f234c0681b9a99f8064d195 [sys.catalog]: This master's current role is: LEADER
I20260620 23:41:31.208155 23451 sys_catalog.cc:455] T 00000000000000000000000000000000 P df9835b6a1eb410a80c2f0fe4201ee3e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2431d6296f234c0681b9a99f8064d195" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2431d6296f234c0681b9a99f8064d195" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39387 } } peers { permanent_uuid: "0a5d1da7ecb34d4688d0fda3b6752345" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39781 } } peers { permanent_uuid: "df9835b6a1eb410a80c2f0fe4201ee3e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45181 } } }
I20260620 23:41:31.210116 23451 sys_catalog.cc:458] T 00000000000000000000000000000000 P df9835b6a1eb410a80c2f0fe4201ee3e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:41:31.210682 23472 catalog_manager.cc:1347] Generated new cluster ID: 424c6053bb1045d497e6e34f6860eb0d
I20260620 23:41:31.210798 23472 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:41:31.214978 23483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a5d1da7ecb34d4688d0fda3b6752345 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2431d6296f234c0681b9a99f8064d195" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2431d6296f234c0681b9a99f8064d195" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39387 } } peers { permanent_uuid: "0a5d1da7ecb34d4688d0fda3b6752345" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39781 } } peers { permanent_uuid: "df9835b6a1eb410a80c2f0fe4201ee3e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45181 } } }
I20260620 23:41:31.215277 23483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a5d1da7ecb34d4688d0fda3b6752345 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:41:31.215425 23492 mvcc.cc:205] Tried to move back new op lower bound from 7299067458184708096 to 7299067457704734720. Current Snapshot: MvccSnapshot[applied={T|T < 7299067458184708096}]
I20260620 23:41:31.254053 23502 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 23:41:31.274134 23472 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:41:31.276140 23472 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:41:31.308024 23472 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2431d6296f234c0681b9a99f8064d195: Generated new TSK 0
I20260620 23:41:31.309408 23472 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 23:41:31.401803 23442 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:41:31.385580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36850 (local address 127.0.97.62:39387)
0620 23:41:31.385802 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:31.385809 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:31.399641 (+ 13832us) server_negotiation.cc:407] Connection header received
0620 23:41:31.399985 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:31.399993 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:31.400101 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:31.400190 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:31.401488 (+ 1298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36850: BlockingRecv error: recv got EOF from 127.0.0.1:36850 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260620 23:41:31.401866 23443 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:41:31.391655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60378 (local address 127.0.97.60:45181)
0620 23:41:31.391928 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:31.391948 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:31.400052 (+ 8104us) server_negotiation.cc:407] Connection header received
0620 23:41:31.400293 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:31.400298 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:31.400378 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:31.400464 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:31.401551 (+ 1087us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60378: BlockingRecv error: recv got EOF from 127.0.0.1:60378 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
W20260620 23:41:31.403020 23366 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:41:31.385959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50190 (local address 127.0.97.61:39781)
0620 23:41:31.386216 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:31.386225 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:31.399642 (+ 13417us) server_negotiation.cc:407] Connection header received
0620 23:41:31.400245 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:31.400252 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:31.400341 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:31.400416 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:31.402771 (+ 2355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50190: BlockingRecv error: recv got EOF from 127.0.0.1:50190 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260620 23:41:31.421408 23366 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:41:31.408815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50194 (local address 127.0.97.61:39781)
0620 23:41:31.408995 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:31.409001 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:31.419615 (+ 10614us) server_negotiation.cc:407] Connection header received
0620 23:41:31.419853 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:31.419859 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:31.420010 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:31.420149 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:31.421062 (+ 913us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50194: BlockingRecv error: recv got EOF from 127.0.0.1:50194 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260620 23:41:31.433300 23366 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:41:31.425948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50196 (local address 127.0.97.61:39781)
0620 23:41:31.426141 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:31.426148 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:31.431667 (+ 5519us) server_negotiation.cc:407] Connection header received
0620 23:41:31.431881 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:31.431887 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:31.431989 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:31.432121 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:31.433050 (+ 929us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50196: BlockingRecv error: recv got EOF from 127.0.0.1:50196 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260620 23:41:31.440629 23366 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:41:31.438407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50206 (local address 127.0.97.61:39781)
0620 23:41:31.438595 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:31.438601 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:31.439141 (+ 540us) server_negotiation.cc:407] Connection header received
0620 23:41:31.439337 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:31.439343 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:31.439452 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:31.439567 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:31.440369 (+ 802us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50206: BlockingRecv error: recv got EOF from 127.0.0.1:50206 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260620 23:41:31.456074 23366 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:41:31.453586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50212 (local address 127.0.97.61:39781)
0620 23:41:31.453845 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:31.453852 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:31.454459 (+ 607us) server_negotiation.cc:407] Connection header received
0620 23:41:31.454745 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:31.454751 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:31.454851 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:31.454998 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:31.455836 (+ 838us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50212: BlockingRecv error: recv got EOF from 127.0.0.1:50212 (error 108)
Metrics: {"server-negotiator.queue_time_us":130}
W20260620 23:41:31.470407 23366 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:41:31.468062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50224 (local address 127.0.97.61:39781)
0620 23:41:31.468269 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:31.468277 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:31.468876 (+ 599us) server_negotiation.cc:407] Connection header received
0620 23:41:31.469079 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:31.469085 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:31.469185 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:31.469302 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:31.470162 (+ 860us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50224: BlockingRecv error: recv got EOF from 127.0.0.1:50224 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260620 23:41:31.495362 23366 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:41:31.492930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50230 (local address 127.0.97.61:39781)
0620 23:41:31.493151 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:31.493158 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:31.493676 (+ 518us) server_negotiation.cc:407] Connection header received
0620 23:41:31.493888 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:31.493896 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:31.493994 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:31.494105 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:31.495103 (+ 998us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50230: BlockingRecv error: recv got EOF from 127.0.0.1:50230 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260620 23:41:31.538699 23366 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:41:31.536193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50236 (local address 127.0.97.61:39781)
0620 23:41:31.536433 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:31.536440 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:31.537064 (+ 624us) server_negotiation.cc:407] Connection header received
0620 23:41:31.537306 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:31.537312 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:31.537418 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:31.537534 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:31.538415 (+ 881us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50236: BlockingRecv error: recv got EOF from 127.0.0.1:50236 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260620 23:41:31.614892 23366 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:41:31.612506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50238 (local address 127.0.97.61:39781)
0620 23:41:31.612703 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:31.612710 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:31.613325 (+ 615us) server_negotiation.cc:407] Connection header received
0620 23:41:31.613527 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:31.613534 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:31.613636 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:31.613754 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:31.614656 (+ 902us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50238: BlockingRecv error: recv got EOF from 127.0.0.1:50238 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260620 23:41:31.755046 23366 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:41:31.752791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50248 (local address 127.0.97.61:39781)
0620 23:41:31.753039 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:31.753047 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:31.753360 (+ 313us) server_negotiation.cc:407] Connection header received
0620 23:41:31.753602 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:31.753606 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:31.753708 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:31.753860 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:31.754731 (+ 871us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50248: BlockingRecv error: recv got EOF from 127.0.0.1:50248 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260620 23:41:32.016696 23366 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:41:32.014759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50258 (local address 127.0.97.61:39781)
0620 23:41:32.015019 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:32.015024 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:32.015185 (+ 161us) server_negotiation.cc:407] Connection header received
0620 23:41:32.015405 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:32.015410 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:32.015604 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:32.015724 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:41:32.016428 (+ 704us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50258: BlockingRecv error: recv got EOF from 127.0.0.1:50258 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 23:41:32.122225 23481 catalog_manager.cc:1259] Loaded cluster ID: 424c6053bb1045d497e6e34f6860eb0d
I20260620 23:41:32.122341 23481 catalog_manager.cc:1552] T 00000000000000000000000000000000 P df9835b6a1eb410a80c2f0fe4201ee3e: loading cluster ID for follower catalog manager: success
I20260620 23:41:32.125123 23481 catalog_manager.cc:1574] T 00000000000000000000000000000000 P df9835b6a1eb410a80c2f0fe4201ee3e: acquiring CA information for follower catalog manager: success
I20260620 23:41:32.127175 23481 catalog_manager.cc:1602] T 00000000000000000000000000000000 P df9835b6a1eb410a80c2f0fe4201ee3e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 23:41:32.258427 23502 catalog_manager.cc:1259] Loaded cluster ID: 424c6053bb1045d497e6e34f6860eb0d
I20260620 23:41:32.258605 23502 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0a5d1da7ecb34d4688d0fda3b6752345: loading cluster ID for follower catalog manager: success
I20260620 23:41:32.261327 23502 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0a5d1da7ecb34d4688d0fda3b6752345: acquiring CA information for follower catalog manager: success
I20260620 23:41:32.263362 23502 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0a5d1da7ecb34d4688d0fda3b6752345: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 23:41:32.543043 23569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:32.533307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50272 (local address 127.0.97.61:39781)
0620 23:41:32.535568 (+ 2261us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:32.535575 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:32.535905 (+ 330us) server_negotiation.cc:407] Connection header received
0620 23:41:32.536141 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:32.536146 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:32.536323 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:32.536439 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:41:32.537032 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:32.537997 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:32.540161 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:32.540687 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:32.540851 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:32.541505 (+ 654us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:41:32.542132 (+ 627us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:41:32.542246 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 23:41:32.542453 (+ 207us) server_negotiation.cc:299] Negotiation successful
0620 23:41:32.542539 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2124,"thread_start_us":1880,"threads_started":1}
I20260620 23:41:32.558053 23575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:32.549282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60392 (local address 127.0.97.60:45181)
0620 23:41:32.551333 (+ 2051us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:32.551340 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:32.551362 (+ 22us) server_negotiation.cc:407] Connection header received
0620 23:41:32.551507 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:32.551513 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:32.551733 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:32.551926 (+ 193us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:41:32.552348 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:32.553581 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:32.555147 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:32.555785 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:32.555939 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:32.556417 (+ 478us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:41:32.557191 (+ 774us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:41:32.557358 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 23:41:32.557592 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 23:41:32.557688 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1915,"thread_start_us":1796,"threads_started":1}
I20260620 23:41:32.576639 23575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:32.564734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60398 (local address 127.0.97.60:45181)
0620 23:41:32.564913 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:32.564919 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:32.569961 (+ 5042us) server_negotiation.cc:407] Connection header received
0620 23:41:32.570233 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:32.570240 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:32.570349 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:32.570420 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:41:32.571175 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:32.572270 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:32.574407 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:32.575058 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:32.575154 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:32.575832 (+ 678us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:41:32.575977 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:41:32.576036 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 23:41:32.576237 (+ 201us) server_negotiation.cc:299] Negotiation successful
0620 23:41:32.576322 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 23:41:32.584014 23569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:32.564533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50276 (local address 127.0.97.61:39781)
0620 23:41:32.564724 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:32.564734 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:32.577340 (+ 12606us) server_negotiation.cc:407] Connection header received
0620 23:41:32.577667 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:32.577675 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:32.577770 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:32.577832 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:41:32.578419 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:32.579587 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:32.581761 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:32.582442 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:32.582546 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:32.583168 (+ 622us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:41:32.583306 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:41:32.583367 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 23:41:32.583613 (+ 246us) server_negotiation.cc:299] Negotiation successful
0620 23:41:32.583698 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 23:41:32.587561 23581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:41:32.564723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36852 (local address 127.0.97.62:39387)
0620 23:41:32.579365 (+ 14642us) server_negotiation.cc:206] Beginning negotiation
0620 23:41:32.579374 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:41:32.579406 (+ 32us) server_negotiation.cc:407] Connection header received
0620 23:41:32.579553 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:41:32.579562 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:41:32.579823 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:41:32.581192 (+ 1369us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:41:32.581423 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:32.582871 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:32.585340 (+ 2469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:41:32.585848 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:41:32.585990 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:41:32.586479 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:41:32.586818 (+ 339us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:41:32.586952 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 23:41:32.587152 (+ 200us) server_negotiation.cc:299] Negotiation successful
0620 23:41:32.587241 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14462,"thread_start_us":15339,"threads_started":1}
I20260620 23:41:32.592274 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 23220
I20260620 23:41:32.593357 23282 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:41:32.706369 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 23220
I20260620 23:41:32.728609 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 23294
W20260620 23:41:32.728601 23310 connection.cc:535] server connection from 127.0.0.1:50158 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 23:41:32.729849 23356 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:41:32.837380 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 23294
I20260620 23:41:32.857836 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu pid 23369
I20260620 23:41:32.859156 23431 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:41:32.969952 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskN6eSV7/build/asan/bin/kudu with pid 23369
2026-06-20T23:41:32Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5030 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9644 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (367423 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST